Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add nmf #33

Closed
wants to merge 4 commits into from
Closed

Add nmf #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
64daba2
added NMFTM model, NMF default preprocessing steps
Jul 16, 2024
a687899
Merge remote-tracking branch 'origin/develop' into add_NMF
Jul 18, 2024
e85772b
added theta and beta to NMFTM.py
Jul 18, 2024
a94ff80
add **args for model validation
AnFreTh Jul 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
314 changes: 314 additions & 0 deletions examples/NMF_TM.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,314 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Import and load dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you have not installed the package and work inside the repository, use the following two lines of code to make the example word:\n",
"\n",
"import sys\n",
"sys.path.append(\"..\")"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"c:\\Users\\kantg\\Documents\\venvs\\stream_topic\\lib\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n",
"c:\\Users\\kantg\\Documents\\venvs\\stream_topic\\lib\\site-packages\\dash\\_jupyter.py:28: DeprecationWarning: The `ipykernel.comm.Comm` class has been deprecated. Please use the `comm` module instead.For creating comms, use the function `from comm import create_comm`.\n",
" _dash_comm = Comm(target_name=\"dash\")\n"
]
}
],
"source": [
"from stream.utils import TMDataset\n",
"from stream.models import NMFTM\n",
"\n",
"dataset = TMDataset()\n",
"dataset.fetch_dataset(\"20NewsGroup\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'en'"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset.language"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Preprocessing documents: 100%|██████████| 2500/2500 [00:15<00:00, 159.37it/s]\n"
]
}
],
"source": [
"dataset.preprocess(model_type=\"NMF\")"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['anyone', 'car', 'car', 'anyone', 'car', 'car']\n"
]
}
],
"source": [
"print(dataset.dataframe.iloc[0][\"tokens\"])"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"anyone car car anyone car car\n"
]
}
],
"source": [
"print(dataset.dataframe.iloc[0][\"text\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Train the model\n",
"If embeddings for the model have been created before, they will not be created again for faster computation"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Index(['predictions', 'text'], dtype='object')\n",
"0 0\n",
"1 1\n",
"2 2\n",
"3 3\n",
"4 4\n",
"5 5\n",
"6 6\n",
"7 7\n",
"8 8\n",
"9 9\n",
"10 10\n",
"11 11\n",
"12 12\n",
"13 13\n",
"14 14\n",
"15 15\n",
"16 16\n",
"17 17\n",
"18 18\n",
"19 19\n",
"Name: predictions, dtype: object\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"c:\\Users\\kantg\\Documents\\venvs\\stream_topic\\lib\\site-packages\\sklearn\\decomposition\\_nmf.py:1692: ConvergenceWarning: Maximum number of iterations 200 reached. Increase it to improve convergence.\n",
" warnings.warn(\n"
]
}
],
"source": [
"model = NMFTM() \n",
"model.fit(dataset) \n",
"topics = model.get_topics()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{0: [('like', 0.01778463406770215), ('think', 0.012829159567616088), ('said', 0.012061456572159825), ('time', 0.011975083662851407), ('say', 0.011244012761108077), ('karina', 0.011057991924920883), ('know', 0.01037644101572665), ('patient', 0.010291494449921423), ('really', 0.009794810358940963), ('people', 0.009727681240959353)], 1: [('gun', 0.015370704284626373), ('right', 0.012566082217764634), ('people', 0.012237137369128424), ('law', 0.011830632323819163), ('state', 0.010004811114279154), ('government', 0.009861724786197427), ('weapon', 0.009098744673566384), ('think', 0.009042216355081708), ('child', 0.009020639481129048), ('moral', 0.008806606930614712)], 2: [('car', 0.08547279137816774), ('engine', 0.026084546528355568), ('auto', 0.023390567847810746), ('speed', 0.021905926345329972), ('mustang', 0.021448376516026407), ('model', 0.020022628479791194), ('insurance', 0.017987015673823905), ('rec', 0.017855635847995006), ('mph', 0.017546260554605585), ('driving', 0.01647549016193394)], 3: [('key', 0.0711181764646433), ('chip', 0.04872475302843556), ('clipper', 0.04449663019737206), ('encryption', 0.04431189738923937), ('phone', 0.03194578648991676), ('government', 0.026828201251758076), ('escrow', 0.02638219031155782), ('security', 0.025245330120515233), ('algorithm', 0.023951340419453224), ('crypto', 0.020078680376742326)], 4: [('email', 0.11411332717909153), ('address', 0.08182383829943436), ('thanks', 0.04702941679050788), ('sensing', 0.039687611991912744), ('data', 0.0390442035259747), ('remote', 0.03441729097586886), ('internet', 0.032809246482209294), ('looking', 0.028011508899594046), ('respond', 0.024841272738216097), ('cable', 0.024782727272403767)], 5: [('color', 0.11551412614434456), ('image', 0.08914434155068497), ('palette', 0.05063083781574068), ('dib', 0.04147601961624691), ('field', 0.03996905409656587), ('byte', 0.03710257670518803), ('file', 0.03328526659953093), ('data', 0.03282974696314215), ('bit', 0.03128163978314409), ('structure', 0.027859217705283365)], 6: [('window', 0.16416124364828083), ('font', 0.055199794526827106), ('manager', 0.03879677665833143), ('application', 0.03825484097748764), ('microsoft', 0.03777634491085756), ('mouse', 0.030842737999959114), ('win', 0.030293975019755094), ('mswindows', 0.02784286006583027), ('run', 0.026655607629493313), ('xsizehints', 0.02566927437406902)], 7: [('card', 0.13486708515435922), ('driver', 0.08127385842724676), ('video', 0.05299067623201464), ('diamond', 0.04489078929032737), ('eisa', 0.03933404682956441), ('mode', 0.03825936815485937), ('screen', 0.036389125796352256), ('bit', 0.0360016376570589), ('bus', 0.033868881430123274), ('version', 0.028652693216050344)], 8: [('file', 0.09048184738675315), ('output', 0.057241052796803256), ('image', 0.050870883252299576), ('entry', 0.03962979520686868), ('program', 0.03283285055493243), ('oname', 0.03199974120133377), ('contest', 0.03039975414126708), ('eof', 0.02826877665547774), ('line', 0.027761861855020675), ('remark', 0.026727253837741712)], 9: [('god', 0.04791019455120404), ('jesus', 0.043629194245705924), ('christian', 0.03060803664632826), ('faith', 0.02382182714162914), ('truth', 0.021773693757169203), ('sin', 0.019278971872132407), ('matthew', 0.01870428244122406), ('bible', 0.017830651508727976), ('belief', 0.017410010556364576), ('believe', 0.016696093535856847)], 10: [('israel', 0.0844188690526544), ('arab', 0.07805279186236615), ('israeli', 0.05969867136246096), ('jew', 0.03710517717997269), ('land', 0.030010160224280092), ('lebanon', 0.027675307106527755), ('peace', 0.025260027236933325), ('adam', 0.023630142935362586), ('jewish', 0.023185229355961078), ('lebanese', 0.023011301464906226)], 11: [('comp', 0.10122506032278088), ('edu', 0.100217219546233), ('com', 0.06352934567598183), ('programmer', 0.057930511350415936), ('vote', 0.049942557679589966), ('voting', 0.04528209144851963), ('ballot', 0.04004354520801219), ('multimedia', 0.03730478632414821), ('unmoderated', 0.03615628949208895), ('bug', 0.0348449413538126)], 12: [('thanks', 0.15647383666615644), ('advance', 0.10066275046346125), ('info', 0.06702908108165356), ('information', 0.047311024888572), ('know', 0.04478959781151237), ('help', 0.04431625570347218), ('appreciate', 0.04087199296779546), ('anybody', 0.03533538875907571), ('mail', 0.03240838738305326), ('park', 0.029785993646127152)], 13: [('drive', 0.0860088779066201), ('tape', 0.046283965790203295), ('disk', 0.03609799717447583), ('adaptec', 0.03128132922404724), ('sys', 0.029017704376111987), ('aspi', 0.027491440375105147), ('scsi', 0.025361845586481545), ('problem', 0.02495898517111128), ('floppy', 0.02485183158878599), ('backup', 0.023637091211523603)], 14: [('armenian', 0.057598825151576376), ('turkish', 0.032131837009420065), ('genocide', 0.02800582993985208), ('armenia', 0.02051961473633554), ('xsoviet', 0.019767301263167725), ('muslim', 0.01639321341334644), ('said', 0.015957183829578565), ('azerbaijani', 0.014974059965554983), ('russian', 0.014427111069311285), ('woman', 0.013866270296009361)], 15: [('bike', 0.15224547932952118), ('turn', 0.0379058115368426), ('countersteering', 0.03691259005481579), ('jerk', 0.036320675925634635), ('karr', 0.0341463260226484), ('ride', 0.03284567168232858), ('left', 0.032153706763669916), ('rider', 0.02544647559221203), ('tire', 0.0238877612235117), ('push', 0.023416220445259005)], 16: [('scsi', 0.11801783984948415), ('controller', 0.05264304440987154), ('adaptec', 0.04790270634476712), ('tape', 0.046467576097488146), ('ide', 0.042219126170357946), ('aspi', 0.04051041128611287), ('drive', 0.0404714789165466), ('sys', 0.03997079935417774), ('disk', 0.03314909140633576), ('driver', 0.03311784664806106)], 17: [('game', 0.08096242550633438), ('team', 0.05472424746207398), ('player', 0.03676323332933206), ('season', 0.03162261362437942), ('hockey', 0.022442456959776216), ('play', 0.02230190314802031), ('league', 0.02166557405027535), ('playoff', 0.021421920608806832), ('goal', 0.020911728364508966), ('win', 0.018135914352210746)], 18: [('space', 0.05075331359889219), ('nasa', 0.03647303482357237), ('shuttle', 0.03110742693830266), ('mission', 0.030862339243093725), ('orbit', 0.025625338799235044), ('satellite', 0.022837781184377254), ('astronaut', 0.021249455517618376), ('launch', 0.020223545986956748), ('earth', 0.019389699200812224), ('propulsion', 0.018424070928276942)], 19: [('offer', 0.07577553740915191), ('modem', 0.04663629294793002), ('simms', 0.043366269731906125), ('sale', 0.0385353461917897), ('shipping', 0.036138558109921774), ('speaker', 0.03300633305368759), ('apple', 0.03257386591132039), ('new', 0.03007684200750244), ('asking', 0.029923379801949306), ('best', 0.029278807568096324)]}\n"
]
}
],
"source": [
"print(topics)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Evluate your model. \n",
"Use all metrics available either in octis or the ExpandedTM metrics, ISIM, INT, Expressivity, Embedding_Coherence, Embedding_Topic_Diversity and classical NPMI"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"from stream.metrics import NPMI\n",
"metric = NPMI(dataset)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"-1.0\n"
]
}
],
"source": [
"topics = model.get_topics()\n",
"score = metric.score(topics)\n",
"print(score)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([ 2, 4, 12, ..., 11, 9, 0], dtype=int64)"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.predict(dataset.texts)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Visualize your fit model\n",
"Use a port that is not already in use. default is 8050"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <iframe\n",
" width=\"100%\"\n",
" height=\"650\"\n",
" src=\"http://127.0.0.1:8051/\"\n",
" frameborder=\"0\"\n",
" allowfullscreen\n",
" \n",
" ></iframe>\n",
" "
],
"text/plain": [
"<IPython.lib.display.IFrame at 0x24997aa9540>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from stream.visuals import visualize_topic_model, visualize_topics\n",
"visualize_topic_model(model, dataset=dataset, port=8051)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "STREAM_venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.11"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
3 changes: 3 additions & 0 deletions stream/models/KmeansTM.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,9 @@ def fit(
["predictions"], as_index=False
).agg({"text": " ".join})

print('########')
print(docs_per_topic)

tfidf, count = c_tf_idf(
docs_per_topic["text"].values, m=len(self.dataframe)
)
Expand Down
Loading