From ddf4070c9ae9188af4e263c12cf1aab5dfe7b18b Mon Sep 17 00:00:00 2001 From: Alex Garel Date: Fri, 26 Jan 2024 17:39:14 +0100 Subject: [PATCH] docs: matomo perf tunning again (#300) --------- Co-authored-by: Charles Nepote --- docs/img/matomo-alerts.png | Bin 0 -> 34151 bytes docs/matomo.md | 102 +++++++++- docs/munin.md | 4 +- docs/reports/2024-01-matomo-perf-tunning | 230 +++++++++++++++++++++++ 4 files changed, 328 insertions(+), 8 deletions(-) create mode 100644 docs/img/matomo-alerts.png create mode 100644 docs/reports/2024-01-matomo-perf-tunning diff --git a/docs/img/matomo-alerts.png b/docs/img/matomo-alerts.png new file mode 100644 index 0000000000000000000000000000000000000000..409ce2ef57fbec01fd0df258e6449c4a2d1188c7 GIT binary patch literal 34151 zcmeFZbyQXD*EM`lK)SnIDJi8Sln_Pfl#o_HrMsm;1d$Mt5D)=rMCmR?q*EHCq`To= z`;Ol;#y7tAkLRB^zVR{c(8D?V?0v;rbImp96{f1JfR96igF>P3@7|G9N1@Q&P$)Dt zEKGQX(29lz{)g@;b5|1!e!Q^EL*eh#PV%}=8n%y|TumG-P>*eFpIPuanmJfl*f?6+ zI;~% zwps>957m?Y;V1liB-}0-a+3qOx=Le}CB< z&D{_q%MyJ>Zw)>NzeWC;b7-;Sa1H(EN6{z+J4Hl)!`#9m%|OhXikgN??|_K-?;o|1 z_iC!o^YVD$bN}^`_G}4K30tX@xVLY=|0ekNt<-V0xG70&D%#qEA>kqaUOqg#UZHft zOJ4k+w-LulVtkzrK zl}eYF`}^m^e8PV|tL}Jzj`2h^Lq)Vl-|C5 zE9$gJ1aAmA-d|^BW=2-b{QP`TQPIui!Tg@gdz1-@i4?j+TXR3`54S9aM@Aa@`nE61 zsmT3d5)-4wrMwzwa8==p^#H&9OfyOD@mPi9@YiRw28M>4b8n1f?&aQ3HvMb57Fo3) ztx0vJNWOUR`4KY*2PW!$R@S8)&G3$n`^~MbETW=RlvfSvE1j3^mU`Lmu)UU8uJkz- z7k)bI*Cl;UC-kV}K=s8^-B?##cjgBVPE3Bn3cV8cOapm7j21lKPL}Ikew#{|Y`w~7regw^R92iyq%*;%6 zlFP!v6qI}q1HXQCsr^omI+zY&?1ty((JNi_4pGi*?&>0(n3z}`Dr6*;&@}fw*=+O7 z>HGE#OaJTBTR}MFiq$@Ut)KLhSP%cUp&`oc`Jb{a;pCe`M!sneYdpAqXyxmX*Ldz< z)l9BCKf4j}@?~1uW{-=2oSa3+@9#02EtE#}!^7pJvVSf4bQRT{%Wf2W5AojK4_@la zT6jCg`KfWIM^;YrwUd*RYK6mGFG0XxD_Ke{GjXcIb)8{tvMve+FkK~8YTaP6z_4m# zz3zPG?DV*MU_idcW0&9O#7$IOoGY%%>rfCApC-`aM|>Jghwc|^@`8so;R{`p^tc8=4lZ}}c-2QxhN_cwhkxzGF@wT;H@;yHl zk&%&k_3D*hI61crS6sU5`UFP7a%?OaotOjq`Png_jI1m%E$s^~txxZ5#w!p060NO_Ml424q*sTsL063;WFD) zbwl|P9~2PzDpglkr>+pUm5*#AijH-3{&5)!ClxpfFAC@3s@(lM%QY8)lk4i- z^Lc5mTL;44O;gWQEv)*z;3GEpf6d2r)<<+^_R z(IXC+&R0oEc>4OIrhJo$*Pj$Q%qePStKJf*^W0yXh6y}?)s4I;|9~02_Uz~hos<_9 zCMKqxgM$q04kAiQB&IcWb{5nhdP6MS*xY>n{5cvAFRxmik9S~T;8b%Msi^ZGva^F( z!cv>@qnhSaRxU0aNU#litD|W-Ibol)avSUGQR3p_i@&~bBijoe3<Qy1TmLYZd+S((Fi{c{iyVgg^39TA+4dPBwY`I0PmE{nDjb2r5k?lU^B`5O7X|(elu!iUJa2nYzU@bH?an?mf5cB~E#4(i+6 zxrUrzNh`?6T)3~Kr1trd5)3~Ya?{B-T!~;|A8s#jx8pE(e}1ge+}76A-%kc>oL@ko z0Uo2s``G1VyGvTsWrgaKcD`Qrvk!{)k;&<6qnTe+NRSoTbD3$$y+n2Q&8t^eOigFy zEG-i@wh4*Wn@X+v0yVkFX@#GK4N25a5N;aQMmUW6!l&h*!`rVv zGdz9zv^h!4F?P^bPA+h9F#p{v#Z1$+u?odLc6N5`%aqR51@GRyt6OG6pO>F6VAhIf2|J^8 z0MZDuDE0Jclv5<$=K4H-{P^jwQfz=A>Uck_?mF$QJTR#7Acu4(17RudBfuD1ai>MU z`udv^9y?cbYBCg}T07nfk>_JiWIFfmPx=yu$@ypAd&BkRh+kN^)gJ>Fg;I#5Q^O;~ z!_(x^hfosxr?e*ROVrB^f@=wVQHs=q@Y6X@Ze3XjgWnGYSg}EuT9=tdHv}udGb> zKHjSc-d!D~l))OScGs{nfDuAAccuwBm!@jR0>nP91r1$!Ntf_S+c)!4cmGNUjCJZt z8^6coB)yL`=MLcJI8O%(jihy`rKP1IFJu6WQ2d@uCw992d#t+Vdt>7TJ-tt<>OU$t zcQ-av#cCn;yAS6639b}{^)f{omICe@$8d|aku`*)C>Da462bgx_Y zqfqijOoyDxC$=7Mc6F0hb;@iEAwx${3t2x<*M#_H7{5wWb>Z<{5@aU+GgH5xz*|KM&A9oHRe^UuRP3SFp|DFj|52ITQGa#`L z7agtkVc0()01ZXTalbwgn^ZAxBHeB4H}-D_F?cyn-2W$EFaRsFndUIvIv+_2UcEqw z>J6Qp1jyos0Ay@zteWj!=dp{U5XI0E$E|~WEyOgY>(|MbmX=T@$K39Wn{Q%gp4PG7 zi#J{wE-rUkBDrciO~pawt&I`))*@r8@kdvVT_GBZ(Xf(VfPk2 zB11#NJiRYCkgppD^Yx3Jm$h|ub>BZ3Dxeazd#|1<3RpAfK1YBi*U!(j;?93i2a_JW zJd%=RRQ?kN+R`*h zCE)8@kRo99*L5~_cJk%ufTXriNOV1r1SNjVoroLtdw4?lTra(P^{V`*GEo62P)z%? z)!gGfkBQ5^$6PjR4kZ?K+hBrl9S&IEY`QT>x7JJSQAa{rNz;-D1_p-7?{Cq7{sMDz zxgp@82?+@yiA}%EhS!NZa;DXYB>eL!#2ovTVS%wqXI#&{l?FiI_8U{T=p@{T0FyPt zv`iirCnT{ozx(c=q>cWdkN=h)GCiQ-3hZ84tKfAk$WP|Y)Io0)LK_3E#$ z&)%))#$_odhlhq@t&CL&>EynKgiayeuzhv2F4uL$M_E}JQi?y(1q2H~ytG^Bq~|fJ zrF%ia>ycFsm>zXAXF%a+;&uD&c_k<~(qY-IYo3pmmX>xGn<+T03=>mRQ;T};@gNxs zvel)Z??e>JY{too0m#T=rIi$BB(+_cTq*n`tEjBps$=Ahfs2c~^6LvO)TWBF5ZCoh zc^hyX|DU7sg^=OdvuF3;3Syv&Jojzj=0AUP?k_ZAtgNgocU`3KE|RUStyI^o$sjv2z4@`dvW@x8=NdNvF({rfqy#lz zy$&~uOJb9ID6-35e9A_G4Z-8XBLC3jWSweXjn^TGiHS)?a9dm3ER@38nmMm$wB~JF zE3NuInzlU9MUdohaYnjyGL);Rk$eZp?i(v3LUiIzcjijP+CF}g@{kIo6|uQ_^-a_G zxTKE5f_MA!FlT42VdB4&f1-H3;UWqkLqq-Zi_C%bc@IAGdmY#v93Chb4ZGfH}mEI*JgaT4(S%76>Q0w);iVF4N!-p$+Ce^1L z=_0V0WsQs&VRe?Bz4 zeh~lN96=T8>goR^Qi0s*$CvmFTmx*yDDv!5m2^<`30=!^j9V)W>r0fEO-a zB&MT7LLkK-@@d=lrKugLnk`4a@&Zc&wNY4ev)ov%H?3B#w$@uGF0L@S3zU?U29-|O z7cUV6CMVN>R7oX(xfH!|!>l!eTDQ`P6B#K0`Q&?hd))M290( z3VFK47oZYCs#d6?m}0)$LBK(F|Hf29QE6$15Cud7JVq%xh~q6qCQVaQQz-ZbbPNnY zGkO-%{Or`eCL>VP!c&OYP6TFUT^<`BCuU%XgmL8*U*c`w`|<4?lCxgL$79mc((3Bz zH3Ag^2@XHAUR#&q7DL%a>$jWg)IG!<7XD z1i}@VA+)~0z@;!=>`to?Cm-nUz6p`6=l!jV0H9b|ST4W~-MMqe@9kS^1Y5!tWcBrF zZ|*I>*%hLYRB2xuqo*A0MA)o-Pta-F6mKwpL+3J}%0NOWy zeupT~q7+&^1&in*b^tg*bPF~*R+bd>w7bfoLJKY6_H((ph zZqBxC?C&Fc9q0+NJ~HRyk%NalZksb001b2&XV>8>2u}jYh1{FO%F^<^SzBcI95w+#FsxzN zOvR6W(%fMq`DyeVTb=uw=T~AJW9ycc-%yjmBF_$Ac<)kWA0QIeOIE?1T5K$7yAnhNt>kBUs%sS3XOE`W+H3n3iCLBboJt)|C0pBl6~i13H;0q0HjWei#~!c`HMvU61v}1sIPWWQ&5nkd zseS*3)n!y&U*mU^O7jXK%Ce`KoHwzY>!X<43i=77RAV}IpCb2DmUI>M<#{Cmk8ddR zM~|i(GMG+kEpE)r%uY+5-OF@%d{z~~hq)&^8vhd~Z+q2IBCa}VmE-pXfvyyj?arEn zFXo(syvGc`ep$>oh^i1WO1DG=Nl$GU`Nq3kk(1xrZ(y}t=%IsJ$Le5th+1^YvATI- zf3EwvVk)BqPO$g*uWSpme5$XnhC1c76FlR3clKP)`{Ry8^ln7Wm)kEeC>xfCy70bx z<02}@I5wM==0-;nJh%Or+Xt<{)Y|{oSK&-LYGUFVB^?FI*4wvHeq0=JX<1nz71C!B zJpq^Mzis&D+j_Vx>FP1R#<8_2Q*0#{FzXfFId{0g5E7hjqco}>8M!zcn;0LAt5eX> zR1?;_;j5f{D`;yj`ebI#Guij{BMK*{(#d5BDU50zHNMQ0DbB3IC~VdclcitIc8*UI z)>$vZVlO^i_r`h|>i0H>Q|dhLhQt`{MHH0p9{9&3vPZkEb&oq|AIMzzGw>-4J~};? znJenJNkT%B+LZHLk|UCaKsEq}(V94HD~>(p*3~zx9Bb5{^tCk}%txk5Q=C@Qc#51w zdMiYk{3xhF`N4*HU#!8}aqhx}nzFw`HTY~K@_m+yiCzu~=HAlV<$L5P%$M&XCJb-P zem<{53k&lME}rTB1p z1GnFkdCu=8eXRfVS268$PtSqHrWYFiYc%@&Fu1&;}P8qDT$hf^-I1Gg1~%d z=N7LfmW++%)@N=z2iRQHI+};sJrDwlmZym2+FBb1D=}LM7}52YX&B9GVx2uIE1|{F zJ=lI0Dekj(;Y`dOb6QQUyRS#y?6E=wwJ;I90p(Y?a8+izwV>Q(TbM!8C;Zb#4l03q z+=57aG;?1S+fheULTv|U0~E8cBc ze|O&avTC0x`p%T4ZPvBZc*A2N8iwcVAC(&0ooRbp+R^{j0&K3obwrg!(L+#LnJA)= z&O0L4%1tnK^W{Tb85A4+lz8ZA@WevJWKGLQZ~I0B9$u_oU&LEs(ua~&=}M0KGRdrE zE5AY@*)#J>(hu2A2Gpiptt@?Vv5+AJ2FcwfN{Et=`|FKY-nA~)#!>qFCmU?n@aDBX zP5nr_xCzr=MZh5MW=>|UtKAf+))ZH(vwhq-R`P@q7&&Ij3k!TkLGR31@6+Gd^3LCK zRp$_Y;Ntug>+&Vfyva(TEg^{~mYeUr3+1A$n1CxC9AJm%c z&)Tn_e03kKDqvtq7vvRK5<|UDd`lfjlu+Tu8G(ntd0fJtaK5T(P~n4ne1527l{i2v zoM9VPaA4o*q^tLPt;%>m&DU<{H}!oQ$qTu4sf3Q7xe|nJiC@2tK|}EwWg7oU3M$Ig z$F1WvjM!uiW@6#e%ZcO`u`wARTutD2p!%Bk0J-Ue8gpt`g%f>=s*Koo?;be!X%7$N z^>IVv1ETS^J`nkWMXeZ85BG5f-A4ekIjUnkwTp6>cI5V(6!B z@MA4e7Z$sH6q)x>xn-=%uU&9HX{Hra5swkOAN9zlQ&-%T0|4aIXGfvEXGf=!1=Q)U z2MR0z&Sy^l`8gVHa=FvcPYItpv+bi_BR%8xoBZb7ShACAt{fCU6M|ut&55nw%?N>V zpfcm_J6tapo7Tx?odqrp^@WuvL$!6SE%k0O5|r|}EB2oAvhmOTZ2?n1$}=f9Dygha zt~kz)y^R{@OHIf7Ek|?G^=Uqa9lATL%1JeTg|`V>)_=smSI-RBEpd36?qyOdM96Rh z4VAF}lWb>Q%jNi(lN6f)MJ4oWK!=!Kl-6`L}ThYNC#h0Ucv@E|&~dpYQ2oIkTd>Soz~kpThaW zakYVdbcjL%_WgEr(w@&nNac3aNMvAb(A1q2W7F$>351kW?mWpD-6KHtXMh%RSy7G} z{i%mbQ_GW~xR1LG=j?jd?B$>7zccXG0Pg$5dxIbH24~>3)6uVVXX#UBdblu(hhKfBpYd=tk^dV1&D1qR)q*6c6GqIMNpg+b zr=Ih4;=8o|kXekU67IhGK6l`=R~EjRpLjhe#FJE4OGx1bN5SIoNA{D8j9ddf`Lg=u z*R1`J@Y=s1bS>M6P(dX{ErQQTnUVF9Pb2`WqFfE!7zN+wxH?{&GZuShSAuHdy*-M!6>=>1|1l7$tOTmM;mbgQe~70;wDwG@@wUiomXqvCmh`ip?F2l@|Wu%371@jy%J z5n-;)cUomG$DQ3Bh>ekpJvov#FFzS#3#lEgp93~;m*bmiFmQ-h7W%fE-4 zaOUK|9a{Br5g&$9|gUw;B>8=eU!^eTj$G$K>D5uR)Mw>^rOL>rU@a zZu;K90AyZtPjQ7$CNqn=&UuPhmgSx;q-f=568uH3DyWf#cDSpEqa$CwNT~EX2Cy>= z3bts=Cdf$l@s^ef)e5zFTlj%tQn+*@P(cn2l@cspU!eRfb7`XX8~%$>8FMF|GfB^f z0xRxYxWMj_nOI3PJku9$DxZcv#pigQA3hSxFBL-P9Kr* z^GaajYjEjwZ*E@Wp;PDPj_n)LiSQjVxOX?{votxoec{&EvaaY)!$Y|*=#L3+VWm-N z_phq;d?O;q$KO2KebQ1yqaBa(D=l@ZFSna~TX}x|&E|p5BN7QzH*EZ*!*xNIl^cQL z*W+$HYwu{6)7Fmf<$v~!chPcT+&=zOz**mqir3;jzK21X0%bJZCe68gT1#89Y#i<7_T!V9QWbiEe^ z6)5y>o8v8xlV0b^oj#e%RV>ujk(W)6UdO1DV?kphY594_zQ7`^Kl$rdiX*3WfBItr zmcu#j2u1-piwgh9sE)k63tgV_TF8b{^4aG~_p_SZ+Y=KkJY2F|z6r`ecK7g=e75>g z>2R<6lW#2wk{6mGjUG(KU}59x(Aa4Gl*#ELLT&g`p}(UUxV0l7m}7ulDej)SQBq>A zn3_3|Z+Dy`-8=bBc_Az31*y&PVblJe;WcOb3n0^4@uziNpN7*SZFxc!n6Cfmt9fyAA`vOK@#t*1j7%D3CckaGVTROo z1~vA$CdAQLFG8Eoe9u}&&aW=Kl`jA!bLrMCC^a-C9EKs}-MPkgV70p*s&WSwHN_f+ zgsMlut`e+KqLXXrLpBq^?=<%GZ+7Hr-y$a^U6FFq>^WC9SysNQ)Dpbv%ewbKf&o*y zA>7$_=JQp0@?fw{Q5T1m5I`Vdh~WQJFmI;gW{7#l}iE#S;;m z7D#dMrOjU@kBPTesKrmuUHu;)3u8q^G#0-5%8XoC(#PL-Vi6Y;g}JkaHaY5AY&OaK z;loE3US7kTj6M~^^M{gld%fg=3b%njf1jSlOmLKRQ}aF#g9J`%ao=M{rO(IRYDtxL z#GBY*qC4w%$2^Hb*XTz%TkCw1pSN%iA+?W8BkRTwM%tcyvl}h%%ELW*^;Oz=eYyL-lncGY zg+d0y$zwquWNU1Vgh`Ey&yHnsyCrYd;iXo+6XNg{yWmN19w7a7RSuu~UdMI{QnViS z%W3XWQ&YW9s+AI6{}nR)xa)m>CdLzP?vCR}iYavtrB1eZFk%AkgI>B163hl(yLS)u zHMPq-ZXVCqZ45-Lzk2%?<4LY{!p+Xf_fu06H+Q6A&R#iF|nNGWQ5B>0o%Y$TkFkQ8zTV`>i_5M3-Zd}Jw zn?W+l0t1bHwb9$*sq>x9AL*-#_d^K*cN*(t2^-Le-vkNh!!vJHCBr1+XBJ_r#{{NF z&BY1AXZP|>9dTQIgmAC(@b**}b6+*AO1t(FcJRs#2fQceV>q>cZC_+ zneS&|k-<8!(ycMUtzO^_2B{{Jp$jJ5%HC}tbH9Sj{qzJnQlU7liUTKGE+*`6XgF6p zgG~L%q+t|f?J<$hYAvOLtpv0Yt;#Pa{+{{Q9~QoI`C`8MVLtLRySHz1Yk*rb_v2CW z{@;p;&q!JNqP1XgBr~IZ6JZM$7Y2r_q6-Bsvv}y6bVlpDtjb!tq1V0>Ub=XDZI_QW&mX8-93;9-A3{_Z&L=G71V=rA>#~8=DknR3O*?)Ky9sN9v^fJr!>$^4m=LJLN&|_^& zlMM;KzS^B8TWmjjyJdk_|Lc2D^FWj8Vv3|^{Cer`p8;8DmKM+Jj~#^T2TKt^?^Ouo znkY}tW3O}ggCDdm<6O!$v1G6i5n~7eeIs0RYH10ZkdUyvyxjPBZxxZ5BZ>^@*~YnK zTNi5TZ(uQR{NSsbf-d&Ajt;pf21$NsOtV;+Qu65ffmSjDQKH04f|3Ii1<=v??TW3R zAIoRN0R7e==tB9aIMQe1g8b+NbcW#P@Cy#c1mPr>t*vd#*JC13hlPQ1EzRq2OR>rK zaE`L{>Bv@1*Eux#K%dNvhldA@4OvqN9<_gtMixbGC#d|UKn(_MsJnNA?#iW#f`FV( z+J|o7lNR@?YBmuf%XsyQ$kNK{7S?l*-9IKrJ4=@R+0p71puCZR-nX9K`oh+MtYM6# z`xZL%*VPq+@o0tdKsbOxG{A_A1|;r&wY5^fo!xUfYuS9{6~vigO}#X0cXr|qeSD-_ z2Qq8T2e+}%K$HfG6chM%Xllv8<)rLBXp(_~l}f-=u3)jrm&U6^o<4gv`Gk&fCUFIN z_-;o#IlV<7C*nS1BqhBF-CWS$q%|)X8rZh#BnWa&Z*w|PI~4Sb zL4|b@u@-=M5ru+5MFio9G#z$GGw627oUhqV&?zNey9GL5En(a$?_*(D7Z{*O-1#$b z&wd82}4@%JKZ@9=c1>*~tGzJ=Uy^!)cAaZeWfi;i*>NMyfXpuo}d!;>s{)YCZ zJB+O5WUVAL2sc69&7Du^>+6g3Jv%!)4`yEQfn+0t>djCIXaQI`I5@hT5uKFN`UL&j z+8PyjBs^_{@3jK~jp%B49#&J)i#z#2;Na3JhyXF5F^DSRDW9i51vf~MX$y{{s($TJ~~KB|g&h{}?Ifx!+aTu}cT{C2-s z|C;??ce*?-G@jps6bMx6>Ji)f`wEk_-qSNP{U0q$em#HaB?a?3ZYc^w4FprJU zV2%cr6A}*KaV@*wqqRmc7P{qeU%7(xrxV1TxuElmtgMz@idW~?AKbouyFj}N=2`Qo z;77C$9{6Nn+o40Q=`OLjmuFBxQh&~{j)RNalptV6L`D`E&#TY1*Ct|E;ee5nl7h6M zLy1{qYkwF4F)HRbUk~npe7!G?(10hO#i{(rAoK_i)_>Lk9WgPQtE($w-fHVYlrEsB zZ2)Zoa@Qb=ii>A1wCLdbTK(rki2_Fq+P5{Mg=e6$K$K#Ukp%G|Z2<`fr;@D28&EeQ zA`l3U(=c{i=iriwxD|j2!S)Z||C`R5otK40=GxO?RjUMptCS$0B>|CwVw1?zVW!%H z8C(E=93>?sa?k}a9eZQs!%4s(k$vE`zh(%Q1}2a?G(&O-gy7=_B4Z+%P}p$cAd=&5 zpQ!USf^pXc;Uq3|AebAv$RB%oRa^Fc2rKtK{`%^>`3wlHNI*f)!p<&_#oSzOKidqk zPtMx<6GSNz!%8P&kbZyDiUJJ+s6fbNuUp2bhHe=c zBq2g_P>{%6ee(kLGmfq8y1{1OF)0mA;%qL7;gIB{Bt-Ei56&0peur89 zD$NmfoR1)q`CYY6<{~X=KNGk(T2|Puiz80qbF!%*V9`z%osx1jJtIRN6ds74F_1_m zNy1&-=}-S%5NL*i%&FLJS_Tm{8f=4VAP7X=EZp4pEbu^MLVAeqpWRcl*YK3JMOza1(>y^l&kRH15+I_Iji2W$>#*W*lW zd(wkI)Dlg^a$F^`kSJ0|b>YGVIS5#>f7~5^fuIu?VtCv#7pT?%T7-hE3g_*84uBS1 zprw;DHBE=b2ztXXn0VX@gR9`nkJ$e)O>0Ad;o2(VZ|9@nzilhFeE z6Fdc4;$7+^&XB(m4K+N0*4fWg>AF(clPj@$xGCl3qEyt>ocJ+!*gyYG{yxvC4gL6f{8nxjKZue*Id6WVy4u3lfvJj}Idv@Icmw_&9od1&7?gzJW-3 zz>eb&SO=4U4hy8KAa>VMYaJLMKRP<%7ZE{ZBcPW_$V~xd%rrP1P$jl24Rs+q#hx8x|gs`c&ow3gcs$N96Cf3=}fd{!5lvWye%ODyBttV5QqJYz) zsyz1>|GnhKm8_dC+{qwp;yeS@JjVMeGZRp4B}jRTJ2^XpsVo?x0uAVxF;FX$b&Q~v zkq79LmYe$mOkDF-%1KE{v8GrA1emW*?|lpk!T|in4;OC{gyf09RZ#jpmt%PjMk=_R z)*x@-O;E+n%*-H6TDrQ!mm=n8 zwXD52KxrqY!*U4)!9^`okv!8R;1Xf$_Cgopqd`=7h#3jOm)0OHSg8OuvkD55026fQ z?%nU7&+!2rcO;3CzM$m04VW*OfF1;!t(?wVbkx+?AR5bDg6$YyQVN?&N376cuDxMwj1GWm5Uf0iqi}8f zowad&fjn5&hsv_-Kj7C8C&LM5Ct_c{LWRk_e!|$HR^#D3R&IasX3t)7^_~XPx|G}2 z4GN%a5d&hT*$|-7%}0?S7XuA@c2X*r`)-5d@*po*IT`QVyC)0reWth#a;wBp&**$t zY80pw-$NB*=ja$vRbv86qiUA|}ID196%7qtEae>PTRK}$6MgVi;Z$aJ&sE4av zF;g7^4in5n7bE%p>S#z)2%+q~d$^Edi@tvCbesU(92ynH*E7dyjTaO*x$g{MCPmtp z0E(B)bxT>xu95m%DbaL3;I)46yM27~uzvcTl=q5qsoNZ81qW%DM zb@$&1<7<4p-vAMFb3n(ap{GY}Gm|?|HTD!@b`O}lmh)Ed!-&^k1t9kG#x;IZvwOxq^S8{4=)kW-2@B5){31d z7nfo&ABDvvXXHsF23SWHmzJ0z2o2=w=+Im;KTkZ|S<-;SK<0UdfHpA)HYi63pFDX2 zY4?hc{dGu3rCXeL0Phq%9TDtEm1YF6QUBp?j2#H@P21$lx|kQ)_Dl5!|Gs|7rmQQ$&7PL%)$F8hq4ZI^6rNFVl+d-@gw6&WmxgkO53U z9Sa~J_r&A_|4u}4&_1hW;Ry+cG6Y0fobYD|dv0WR6--kqlXh6d4qusU_=G|aAf~=9?0&A)#KgZv)q#^~3t{lS-s)cm00Mt>!l z_Y~Bah*&pPksWaUoAuMI4x{!8xy>=!3clRlP<5CqY3}?cAvp@Z{=v3LivgE13kKg?->$& z!88n&5GMF_GE+*eftx&lNSUU|We+CEv088aby^<>shFG)Bio5;YQ;<^V-5@+aG$_d3j$p(5GLxQj8)*c^xff zA#!>!nj@~%NctOyA4q)3X0rArly6JV^RAIy3l9IauXE!i=Pp`^nTLLjxK(@zlwDzrSV&$0eGL^Avo1{#X+ z#gWgK!N3{{VWECv;vx4lNWpNI0&W||h~*<+zYO!${ud2h_i6 z`(4SU36Eu7*!Y)h>rRDXB%(pqyFT=O^w+N$Amo~}Qhjg$lpy9nk2q6MI1700J&jij2;~hc#B#uvNQyxYabT*tbO_}W;wFY$ zpAI0Ds!#Pj!-h18up$WB0wSz*J{!#sT=CO@$`QwhogIJe;XG}PW9@G*j*Ey**$g5Y zLR^$s{3uO9gYBC6SXkB~Tc?l#T&*uab8WWu`}?7|2pB4KOU$uhUNK=QLT1-^vN#0E z0jc@{e$awKKjHclikaEjmp%k>Kjy$(e9OyA0wSl=Ana#j*gqTF+W}zW0*5YQu37xh z=NeHF_j6bx$LagGZ^m{Z*(h`SO>B>2yHBx-=8wP~z>5!rqg##E4;@2cmiC74-?4!} z6%_3eh5Z&6(cP_zSQ5dG#kG`9^!CG-Cqt&NEa5~0MxV38$Ef-n>(v<|<&KC4ePx6c zs4AqOe(0>mnF@m%NG%IBbug;|LU7_An~A~fC-6$-H(QXsI~i1 zs|$YwxLX|!Q5MpA)IZ&YI8jLG_fw_lnVFf9!ndImpm`Hy9_lFpVpwapA?X1YOLO(M zu|3Z`nI-~i;qM>Pwvn^H_2;3X_vtSF=d=}$NVNZu^`ve8eh+bUn3>o% zwqia*^`FCB1nu$ueE{OOqL}|V%;o>TJPf&@{i9Y6KoLBh4*wo_qlQLJKb2!p;b0&C z_i-UGA2^}1rT-t9mZS%FDgFL_mODzB>dd^nKaFXCQT{f!QfM3=NaS{MAOM!3m%tpy zJK49B#niGwy)!*OA1r-#&}3X`(iF_h#`c>$?voQP4o)MW_niXdcoYEl(Qn^|!S;ht zyi5Kj@_t;}GW&wJ^&$BmJ>I`pNree}2=Pp1+@&1&9cM|%z>E%iJ_ynWlaLT8V0R=t_CyaW z#&J#esYtgxDXe+KKMvpraVMebp?VQ;T{G~xxroqkNWKNWUgj=k=GZ30a_Gf;eL8Xh zSQuU-@YO-l&2P~jOTuF~P34&yUkZR`8oZEyr8(rh7`S70IM{_b@Ftv)gQONXwr3?S z&I$@bEFe_d7=52=9>GBtq(Jz87cvCe3*p_t7)n|Gv6nCJY@s}PV7Xb)d`3Di=y)ti_cN{JtwSL1`1{S{L(GFSm1AowTFN$ndJ za{dY~ZU9)W0*Syl&MtL+;t0-YZSHYcFveC^*g~IY+$A~>-4iCoCv=ZeoUf+_fmNp% zaRf?wQUI5rdWB5B^4!0hCsn}eK*Go+S6+3w!1Xfsq14|A6X(%!Z_Y z$ifqg5}0^YuO8FKfDcjFW-JtliZ@Pb;YkAf zR(GL6NE?hteE}*LIGJQRMn$TjH}l@~$_iq+gnr6?oX9-P+y+!n8GU7zJr{WOO1}w> zd#=}90w)=QlK|6UaE$^RgYYhh5gHP)dze#nr)}~s@L6D;N^{?yhiBCrg-VtIFadJR z!t~M-(oq5L`9-K@TL6ZJ!ucdM{RWhBvA&aD+xW22h=E@Ka?uQI*#~eC#TWl-D79o~ zTVKBTLYJ5ta+p8+GIRZb&h`WF64QVa3-F9{^m^6%h^6bD3x`|NfFe;Po|}tRj4`iY z^TW1t^Yk=3+(NQ50-&j;IOpKh7f`FGspsH*%r&Uk;^tBt(K3}S1u%{@7aiyCYr^47 zP!u*oC0zt$X1;W_dQKqt_<&Pq<{hnYyv%Vw0d>=}yb*R8a?}wpItNEbGq4y@XRE!5 z5GuedIDfSB0I#^vF zR8FA12Vwy1>K}KG!8XV5edLs14ZFMto)bD-6wtpyW&?W9{IKJJ2FC{b{&0!Ki&!aE z){Ee4M2K>vEaTBFLOpo!p!W!>5IlNuGT4BF%vjH;r3)eB0_+|E2xqES2Z<~V5dZfG=ZGl z1nm~+5ng!Zz1|7MG<05M+9K&UVZ)k2fP#&83Gr=mYJ9XDX9W5=1SSOHD)d}+p-M8) z+lL*4Abh0N1|DRW^$C2Ys;uu&Wg>)@n}>&WoF}YBWZr@BD}qgIbqM&f)WdVmrR&$o zNE%qf^>(d&zymr9;5tFVT@Y#40rh-m85zJjUtd@kh@Ti>F~aD5*afVGggdz0)cShi zu3xQH9|yw4K^s60!VWPFjrGRKT;6BAxqPWCD3t%&lbpiV@cozi)%*X%ncwdmbfqgau!VSa_kqfwQ}kbtciT)Archd~wHRcV{Q| zkBm}99CO)Y!`;tza9RnF>H&~wVXPE~GCm?lECE9W6W{hNn)Qp+GVXSV1gthVF6-gZ zDeN$4GlwW<_R{!h({*iE%-zk5QqO@3yG~gd>Rx|1jt)*;L%4lpwd#$kE)8U>u^K@A ze_1)H9`0x745(OQ;95DC(i3aT575)rFAMJlh5FF2Ha4*~=a0I|sd=6p6uZs76e*2| z6Ks0mz#AZV^1paN$KP~wrlofsSSboP^npwUs|T8f#N_1cv5J|<2bv*VL$3!}89*ID z0}Qrz5)eX#(3Jp524dXJKY^vrU3;>155m!3Ef|Q+@@{VB1KRYgVE zr0==ml{m#>_w7dryPfYSVAacp^aM>!P354!+XDKN4_lwQUe2Ef5y1u{MGkLc5fgiW z^q=5tgW@McSG|4pxKO}2BFrBGLy%Yi9RJ) z2_f?`G?W90&L^^DEGMJjeju1CKsy3Xa6(STl82)N0%7HGEkjmmhBPjB`*tvp`60+e zKrUAI0XYpNoUy}8p2sRWJdKAI2xi6$W?>|MUKW2_bmj*;00tp zT(|?D(_Nh&kR=F!1_K#r=qAV^&BNyxFCooCDQ`Wf?=&Rf_#H4=|Mff(R01;X-2D8i zKngs7$pXLc^&Utgz|r_(LF*M8n;24bMQ1IXtnt^dz+KBhNc*dCMI>{w(L|_zRJ}IR zC3SrN>F8Ibh2e-|-*X?Nd&w-fcm2~@-BA)8VWW)@0pMc~1GE_i`#&5kj<9m&0I#(` zAPZE_AuF?;q=0xn1N&9|5b2geHx+5c0k?$NmHJ3BPG>^K+&BJY^87Tl^D8Vq@l*wC zF!`qg!3T{@$*yJp_9)0`h7xc*6qZT34IC<$rdw_7@tq?la|IH)`w{?|dn`+c?zNh%eWfIJcuV;25YrTMb@HwKQFgp6;5Nh3S81oN-prLfRK2=8-Eq{HykSS z-fi{EP&4#A0*Qh=AAypi2ryUO&N0YG_@I9N(yjv`DR}Ec?_-ETz$4xIWHx|sK9De4 zKq-KHvKeU|B7NL|fBIIT8N|Bw22N!-p>F>+9!R2)@Nb z>TURJY5XO0bmZ7Nlpl1vVO~g~#VziR!;GB30`VpTVC`+f|Iyx;#$(yG-`-M`3=I@1 zLW7}Dh9W~Dvm(k+hRBqplFUO1O+to32q{CiAxX%T3{hmJG9?t1c}n=Nv!3_;z3<-p z!b9=ukR5ba*JXd-pl)6f6;Vp_p19_fxq1 zv!kF4K6Xf^4g!Y5Pur;(I*D}Bm%cBL)T9Hv0)|KUSJ z`>+_fQP9L9oJK5gCGaX0sL2RA31II-n1vXPOdTk5;2beE3PPfHxUvJo0caWNFU5bT)1Jhh%*GN zEDa^!dH6Pr&d9keWQ>}OfJDSd2w9#l97T_9z+*Hxe!L8~09d9eKWHqUo@O%~2JtpS ztwwWmb1OmTh1|Qh93mf_?B+OES1G7|B>p(4T(F^RI`SnD<;~8XWd_{_u7!z(O9uGj z9Lc9imp{U|D4fxV>JD7{awevOK%?NO!U(1piWw*#C4bQV0k0j9P)T|D!Qt2M zT|2UgR~W)h>HHTfBs?y#NTi{V933(8**2kcxH@dyvrg_*gZwpeodX98oDr|UUvrz7%YryZ=rjW1L?>yhCyr*F7CG73z6i~lnzwXXmxar_ zHV66>V;&?8#NiHiX!3UJ&Mxqm-GhTS2z8C}n7D#a%HkApZW82N`4&43?g8v^Jp9M} zyrzU0T*KoZz;R^>Va8Ee;i;kWHF9_%@K!7v$r#?fNXkGHF)M|-@9RUM+XJ&#`qh#Y zZEX07|7lyB!Nz@;7$Girg+$(Q{`;c_0U4Q3W)5jU6!Z_vqBqar12e^fJi2)usYQ01Tj|1lhm(Z~%B{2M|&vh`>yuWuf-n5v(ZR}O3VTg66 zQRv(DwziV``qic$qswq&kkSVb#7VH0(vDih;Rx7w0LoEtp7rpAJ~7>j3Y^-*5k;V9 zglvu-A`~R8rfB}Vq^Fn&sKvR>EXq2j| zREb>)mZ*_&g@nYJHqlmh#`oSV)qNLilM}3F2i`{diWJQ(L58*;;4M@nlZRI6WA{>d z44*%L4hDw)f%{>oBNCt=aGClZ**TYkzi33${t|WD(cu6~lW7>%=6AX}qBh<v-^i({yO6_eD%ALt1Z#}e4mwtlssWsH;8a_!s1%^id@ zAF{p^*2X9VRzbR<^&wdR&cKk?2r7D|uN0hPII`!+8{$O^O7AxCnQnwzC7gZ~_Jxg5 zg!w{r)A{?xV0(t>m@b_KN2$FOT-7+J=N4|@1H@Mv%ElK?LlJs;UGTrbPCEz$9hQYa?ny3-#KAE}uMw32 zh(~1@BO@C7!_t>jgwo%{i|uosJ-Z4)1eVz-QsJS8KcJ=iH9Z`JMAQrmE;Bb*1wlM% zl6cj^5jg-EO#k*A{xtq6cH3{C`HT?ke2?RD^GZ-{9G^kDhXNa1Hs0Z-eJi2H}+yPfg3xOu3{c<=l z%!L$2O-&8iXDh>wFESa+er&qaDjR*=I85w8Xd}YnYH0gN?r8KJnJ~!&9eh-DA;7tK zdyj~JZCw8zS=T;Kuwh`FzZ$oq&}eIG>qf{kn%oyGyBG=yTkWJ_+t3wFeeiMM(N3N{ z8*AN>tj!PGW9>WTNgJW!x#mzbONU!i#QD$U#SDiBhrvG+wuvmrZYfWMxR)PVnu<=UxsX^DThq%QOj1Qz{ z>&Kd!>S_iN)1sxp9kd7tZbTjnKUqD&@Y<0bo89l!K*_M2=N#mb07Y!6Fs~jaNhgYW z3#>h(){7?7n}g@IsXxonnWhcdV(wH7#uNOV5kM}CH2O8PFYp}9I>G&ZT-pI=eL2br zc}FkBS6jbEf13;8e(8Z2Z|;1MYp@#!>sarO){q(k@jUIv*87rM1@M^{nf zTEbJH--FZf77{vMKEAT<0R&Jg4&J1?&wokgm+$MmPD>5BOa{;6<`uD4z{NETY<)04 zYn#jnxHYh^x7Qb9Fg3*|BF6a-C6?xj*1e^`gw-E_w%Ovsyjy@Z&A~+(hcmHy`6%#q zy-?JnY+ct)<8xSko|UAa;Nu}6B#9rXa2-BC$O%JFCXlFZ0c>Px{TFA)S7@uD3tCYW z@DA~G2I53VK}EWP0P#310sZ|ku$PnLLOMjaLa>u`U`xeq8JK@z$iFw@M8?j7*bYX0V>4B@#7|oV)&uZ@w>HxtuMi zyW5N!Ah>VeK2$?y&`A(4>rT&UV`}bNd|2$!g%X_50y}l5ldGf{lQ+Y$mjQwq@U8JE z>PN=L8W)L5vb`ykM}SJ&r=@BG5?>4+cf8{|KQcfBSIiL_)N|1yn?KZ zIGv+EKy?WDaPx;;2a$J^;gCxpUY7yLBY?f!!i)t$R|}^*Yo=XSE?>UC*lS+&T=#xJ zM7$VM04dOt;7ta^6%`W&2p0Q$Kz@<%zF=Z0r|RnaU}%Tz8-=ly>72yHIyopt@N^m@ z6Ss*V+NnV4RhyegpkzRSI25eTxg)d0&poJ12vLmxjC)WE{%j4$Xm1s&bzQ7MwiT14 zLnkA6dqTAWc_A8TIMT!HH)G{)S+JvkrYb>O-^7AfuVHEq#7epn9nA@=c39qKOwU3D z4{!vf&PlvkI{APmni+E2P$+PFjL<9q6;P0$OZk@83mHog;(Hl%onU-I2f6ppjW=p@ zOvnyxsOjwF(umkm3Z$Cs0|f2UO!IpY;BgL&tZ(jIfmQcy3*@#C9Sch(SI4Inj;OLkfDTi22|kB#Rk zD&>pk6(QO9v7-%TNFUN0pcGsd-0<=WMWW(= z3uH-w;Z+Z1-Xu-3W@o@O7Z2w*`~-J_&8McezBfE8GqVJRD90zvO%UuuxkRDh+`Q|y zfu5dzNGmPYSv>dKzH8NdgeqRJ#`^E$T}~oX#LZqkU4B-#?ZcRvlF(3&M~zZa1+vlF z`%)I7_Fw}DF+k^{NkG;5%#(^SdsH-2;a*!2@M|$jkpb6V&F0ymjxJZr!y(ID7$sgI51L3^FpiPSo!#BW z&bM1OPhunfpJyf=%R_4V8iI;sCMI8#o9NTgX{9obL7Wh-RlZwE!<0`GLBAlHXQnSs zSM`nN&f6VpLRMs^nyqjMNM~<$d-zNf_jSCCFW0RcnySYx6_Y0ky7EepQDw-d6-^d_)gtdT=cWuO-$Q!;a%!L&qcU zKMk^>M4-dupam5;?V|304W?0z_|P$*LZT?CPUW@d_{OP#>#TlP9#dj><__M5use~0 z68kSmDqN_hn&5~su4wu5OV3%QOG5Q6aQjbrl$G34hUh&r_USiHt6yvpQc}vRp`LAi z?E?XhPP^`zP<@-Co^82%KCIH2jp%Z_f)}ULX`qjr4?lcxUH7huzES0RiaUl4nA7W1GIo<6Rkm5Exc zn`K(FI%M0s&b@h_WpBw9ylAT0(H2eY#on|CZk* zyF?b=hzYB32sss8+VTqZGUI5<=qi^S%kQ0S^aFggd|SmP!++jQ78lU$&^8(jM>f$7 z_$x00eNFLb*Z{=uQofZ>&t&cnp>T0t-Jm%m#OfS z=j!lF6)o$NvP=viLV$qUR__BuT4wMJEkD7V@R^J?cmiL3$X-RhC9p(-Fh1%Iiw5oW@X0P2em><%&F+0xPHDwquAdm$u=Cz3@;%5qSXal%5(ov|BQ59Uv z70i{_JaY58J>~R)M?y%;XnMSBd**16mTNyX#k!Hp=2#)eF%y%WJF`WzOClEJA9OvZ zp->jB3PnG-vic8m_^nS7d3X`Fa6LM&_%^PPrK7aTb*u0nHPpW0ZK_f@IJV)$^_QtR zhb_iweS`z06TQ>Ck_}U~t!Y?uZX&~?&E2_JTuC@%jK$Sr)!@P8-`7zz)rTPZh@94 zs!K@o1zN>!pKg9SQ~-9I%x55iMPX?&~qUCZyUOuq5Hpy)j`4T4qQ;i6JhsP8te=%?N7Kp^eu2_fw$N zyaLSV^bh@aki5cmyh+Ldye*JE{3)V|!Igc<^erUD&1#7pF3|uGaJ37v0jSnm=bVc$ zLxuntKod@)k6YSO3i@~e6!AGS8NE7pStuam$aEMYgkbPnf^ww{Udq8ZQISr^1K^_g z5TOqmSBaPOIf#S5{r;m1CItZK3Sufx@KCT#Fvlgj6V$?yyh_5$kmLv!e=851l zDT9NZG)bdCS9J}NRVIjhU*j4?>4CzDDvkyOA1c*8B*qR;FhM`QZHY|M4<_cz)L#jY z!l!6w=;taa6tCsKDJzwy+P7dh9#8qfbgu|CzE=piJVL zL{al$dWuhk#Bnpjpz;^fzhj&DR2>WD(!7n+8b|g>4*ynH)hKCWPF(GA>2m4CUgp5o z6)vpe#U}6E60AeHmdh1T>ecGWT3`GjvqIv@>F$q1U7SaMHZI;TJzX5GUB#-+u$A(_ z|EN>jy^q}o&e#;y*gpwn@vi4&ZFrn&c~mde#_#c)T{b$|Qa|MiteY-Xm~e*+9($!C z+&dCtx5_FuDlj7>y|Pumm6jqo{qb7eZLL?05AQ9xTfBw+xM|pPw}XwKO>TftBy$T0 zI8S*1&>rI@j)eRiC|;Zwk#_!h3tra`FvMU6OPP#Z!UF7p(17xEVD&6yK3O*Pyi|J- zjQ-eKX5h<6U+%)}05^f3(IPIeFxR!H{_WS)kDKVA-7F-u5=ydi^detF+mM2AYs}q6 z#1D2^H*qp04K*Mp@vH&8Sq9A#fil%E^I0TM+uZ`C5f9iHAaW%{A@70A1M}AudXR7^ zRQK$F=pMY<60k$(fd|D<4xhZOVq65g`XwM7BJKBQ;l2$i1a=M|AK9_$-uhbTiz98u zwPWaHmH|XcusbY9(z9nby!+j5)-N*Pj!8o@)IX&tFSS#SY(;MjpO8>Zx9giBcAxL= z+c5|WT4=~>-bZf(CxrF=3Y2#yxh*Tj#;;O|k;kYyZ0;{w91o1NFLQ|)+eit% z7jiXoMznCQVD3BD+44{RF*z}vlYTm^u5l~m0}#Wy^>RH!!UQsYzhz@XLcw#-{SsF~ zXaWr-$W+<9B}M9M$0rtvC#jq{ag`!`t3z3RB`;BvEV-CE&P!_M(&!z@7aw~and!K9 zy%$&PRLo78#V@x`ZDf|cB3*vYUZ)hu?{YGT4JtG=eBU4#9n)mOMKE>f-z!1m9F7Nc zqBag12S;Og7U`P+RHT5XLOC_fZ-{OyNDQw)^cZ6gM|)9ps38OF29_*6boslh1{8iv zp;3lI9=H*6(&M6`Q3gR4gL9@bxJn`6)M4*Uubv)WK>Zz4mI1pNjA8hT)hEZrv9J z7)CDIH!ZV%Rr~qOjO)ecBER-*k9_m)eS6W`bop6o)%c?;oN^v}30H=;U}mxqiSd3Z zecp7HAU{!l$I5U37_MpEGwwY@YMCLFp2X5M;y^7N`pe!229 zG99EA*PJ)ee-7Y`o_u6Dl&fwh9L-bU&&FSD)RMF6?j!FiJY zK=PH+GLrS_i<75ETMsRxYp-!rX6j{)AYqGdra5fJ@b&=5-CqgD^VX1Nm*gpV{!BX4Q{eZ6t>oUTU^=-Ad&#}F0*4> zQ}r`IX12;--3^!%?{Iik6kd1_fMdFvutvPl^Zjp&%k)%kv0ny`$DTt4@nUX~Gmy!! zeEItgyA!zv!{QuYKuTikavP~TFtc0epJRJ7`f|K>AM~nM~CNh|O1$5Qkxlo&G_%Iin32Y<9D3JC6ENSq2h(08Myjh#I~j9&BPK=-b&+-xd{mgk$w?y z(Q?SI{LpyG2Z?&!epNhT(6dR-#^7Y3To9iBIXoPMz)B?9c!qB2mwBKzfozNyYUGxd z7BbcoJ1z?$9rIb$LuqM!Abo0=vmTE8jSbcatc-9(RLi_~n=HOd;Z%N2NE{4_Mf^hU zhJH)c%ILcmgnfCjBB@|d>4h8~+ z^w6!7U)h^Ti>*Z^GqMJiwDpk?nuBwJ!jMO6TY6AY@qya0br+er9-JG9B6U)7cr=fZ zilS4OSz}=qi<|gpy`h)V;SbTfa_h^B!~~B%2^%QF_CB|Ff}bwO>#>>NPNmMeJ8_B6 zo$RTW%{$UQ+df_BRN<&*^zd|&ttgS1+zZ^NK6XF~PFh3p5~x4otGx>>OBK;|o(#z#z~L2uc{ zz>v7{uaJski?Tg)QJw>hO*(~n;Fw^vE%XhHQ|9!aBE)_98{%>h;-qHl1b(lZENFT9 zCPuDoOHkdji1~Gs`@HHKqx&wswZ}j0yV7+(@Rww}qk+La!>>zsG+pZKWD*GwE^k=A zCu};#+S$ysoGWHw`os4dH*T=EOv&E&NG=XwrkwB|<6P5Vs-E#dc!%d~(y2$?mC^D& z8g&B6`cef4^ve?%NHN>tJk8VNoiQT$9IvUaG%aNNY5i2cfb)y2YI!TuF57!nQKhziGE-Dejz0vBX+yqMi5xvc;tT_Yc)vht%SEOdg8-xuf3?;Xr8kWZ)Jb4j>YNwY zqxcNA*{-LV@tB!ge+j!m7CysprB1lYso~x1z`Bz@y&v!ONH9HLCt0O!_4&k%>yzRs z>Y)!a&s;Xn4Ougu9v6<7cSsMD4j)ogbZ(N9-fI%2vAxQYQTf@TbNand@5Kb?vl~5K z)9`9P*Mv5%{<3@B0bz@fsPG$ci*k_z9zv$3#zBiiY3>`-+^NAPD$Eo+FAORUarIVA z5h;#4`(r-=G~BiPjxG~=sd@PI!W{j%?-||&6`aQxl3(*bv`YG1Z82}TOYi6X@p^(X z=<{YQc^_LaFj`YN^Lrp=@#7<*53TdlrLtp|TZ;DX(z1&)=5*~D>lKo=%-i6)3F+K|B>yoa7f9Y!-~IYVTi12;p#I*45QLPd;*-ys4cl zG^x8l5)%N{p|jzBV#QzR;O+D%Wa=lOjz$Bdm>@d{qBH2r*85cmrN%wclS{BleMmHd zmIj@|fElDKgU2_reUZ6@MovgINY^|Wafq`7rIsSY7DDk;E)L|bz*t(+6g0v<;W`lG zOa{AAEkVb=hH4=T<1~qH8iZdEp_JmGjyo0m1q38w=^NEd{p=7b3`|R|{kw8Q%geWa z4YW%BU-~XJ3OSN}VX^~4eK`-oCZKO3VGv3NXnI{1Cj!cyS!cf-$)L?I>}Q%2#83ei zZt2mH^(i5Tl`<-(O@_$xMK+RvoWJ4yHcG?Q+`(ABUxRG$$6F(`k8;)DAJ$z(C1e5{ z6JyP(-2^rw7!FjQCg)dZ9CL{h@SsqZhN&qX&_i^_@qM(Re6yf4ylW;~s6bM2drueN zIc*PR^Rkh|E)3`Ddqf}fNGL>~<1J5uf*o+0*iinv){H;@bo~`dmrK$IX>RM?qo{O> z)naHR!Fww#Vlc^5uP7{9E#l9NGXir9{zw1hiL9kA;svzXcA~2}%+Gk10t2pcc_N12 zt&Y=g@C{&Dr}Xgf$w4}Dv0qtUTfnx(?Nz#o%aE_C*KF=S0S@Oj|L-Vq2%XD?+tU#> zeI8Yt6HWl=b~297lgVAo1{}Wo?LRz4EHBH5%|i0sH+>fjj%MztHbygOcB)xQ^UQmF z?=R0!7=v^O5w`e4mk7Gwc;fePn3MONLvX(M#Ew+s)jAySlj0vAa@gp3dDFWtiMus@ z7w`^?J}9hR9ex3V6xY9?yCmC3{q75SCx)!e5_5azjGU*w@CXh(5j1!;o%55u?W=u^ z8?T5|qdX)yC_6G9G$|=1nW6(eR?ahO_%ESCXL!Qn%KKb}Bt2d+p_SikZB;eZUky=> z`lGI|Quw8xbVO~@%F+_e^t_nHH>8rH2L;UOq`JuUt0U7t^}j=4K?aQWxhpL}3h|ir z6&WUM{f92nsc)kGa+|~()M+<6*2K|&pHFc)1dlutnJ}A2rxHk(V|<$U1Fo67luzZ0 zb21I?56qWCnR&#w3=&-kJdMuU++L0sq}R!&@|U2|z|)`b%noxbJRy8{!^uJadsZwk zpy!-;7n#_;N@tI96I00jwU)@28*%${c{I9qQ)0VdVPs?cGb@P-h0d^FTC?-e;|8R zVJ+t=rAOOyl@)7ymP!>nN%nuVB>#^IoAuw1r~(e@0_PE${X_AJvNVE}7>D-ew&i@g zjq~2cCz_-?RQVJsU;DNAL?6(5(Z4v0f$lfHZ&g zdq3)5!i9f|$H-@aXRSx~X5P(Oj_KWuoiIT*M*4Uy|mt}vsuuRW`X&3I6eCecwS^KEOYwwTO-Yh8@ae(XB;#7Aq k1z-E$FSd4ylJ82iC$xW>kLON*BR{DfJfxJaV0Pty0NNi>D*ylh literal 0 HcmV?d00001 diff --git a/docs/matomo.md b/docs/matomo.md index 2d4a08f8..8c521f48 100644 --- a/docs/matomo.md +++ b/docs/matomo.md @@ -13,13 +13,21 @@ Ask for it to *contact* email. * php7.3-fpm run the matomo software * mariadb is the main database for matomo, some configurations are made for performance (see `/etc/mysql/mariadb.conf.d/90-off-configs.cnf` ) * redis is used to fast track matomo requests (see [Matomo setup for performance, below](#matomo-setup-for-performance)) -* Two systemd timer takes care of putting tracking from redis to mariadb and to consolidate archive reports (see [Matomo setup for performance, below](#matomo-setup-for-performance)) +* different systemd timer takes care of putting tracking from redis to mariadb and to consolidate archive reports (see [Matomo setup for performance, below](#matomo-setup-for-performance)) * prometheus exporters are installed for nginx and mysql (see [Prometheus exporters, below](#prometheus-exporters)) Most of those systemd services should email on failures. Important configuration files are linked to this git repository (see [using git, below](#using-git)) +## Alerts + +You can setup alerts in matomo. In personal alerts. + +We have setup alerts for main website and app if there are 0 visits. + +![alerts settings](./img/matomo-alerts.png "Matomo alerts settings") + ## Site setup @@ -56,15 +64,18 @@ The repository is checked out in /opt/openfoodfacts-infrastructure and specific We setup matomo for performance (our websites requires it) with two main points: -* it does not process archives on incoming requests but instead on a cron job (see `confs/matomo/cront.d/matomo-archive`). +* it does not process archives on incoming requests but instead on a systemd timer (see `confs/matomo/systemd/matomo-archive.{service,timer}`). See also [official doc](https://matomo.org/faq/on-premise/how-to-set-up-auto-archiving-of-your-reports/). -* on incoming update request (on a tracked website being visited), it does not immediately updates the database but goes in redis instead, - then a cron job process redis entries every minute (see `confs/matomo/cront.d/matomo-tracking`). +* on incoming update request (on a tracked website being visited), + it does not immediately updates the database but goes in redis instead, + we configured it with 4 queues. + then are systemd timers jobs to process redis entries every minute (see `confs/matomo/systemd/matomo-tracking@.{timer,service}`). + And we have an instance 0, 1, 2 and 3, to process each queue. See also [official doc](https://matomo.org/faq/on-premise/how-to-configure-matomo-to-handle-unexpected-peak-in-traffic/) -* `MariaDB` has been tuned a bit toward performance (using more memory) see `/etc/mysql/mariadb.conf.d/90-off-configs.cnf` (linked to this repository) +* `MariaDB` has been tuned a bit toward performance (using more memory) see `/etc/mysql/mariadb.conf.d/90-off-configs.cnf` (linked to this repository `confs/matomo/mysql/mariadb.conf.d/90-off-configs.cnf`) * we also tried to avoid "2006 MySQL server has gone away" following https://matomo.org/faq/troubleshooting/faq_183/ -Both cronjob logs to `/var/log/matomo`. +Both tracking and archiving logs to `/var/log/matomo`. See also [2023-12-11 Matomo down](./reports/2023-12-11-matomo-down.md) @@ -90,6 +101,84 @@ Disallow: / ## How to +### How to investigate 0 visit problems + +1. You first have to see if the request made by a browser to matomo returns a `204 No Content` response (use the network panel of developer tools). + + Also verify that the site_id is correct. + +2. If it's ok but you don't see the visit, you may look at real time visits of your website. + If there is no data it may means the tracking jobs are not running correctly. + + You can log on the server and use `mysql matomo_db` to look at the database, + in particular `matomo_log_visit` table contains visit logs. + + Beware that it's a huge table, try to limit your query with something like + `WHERE idsite=5 AND visit_last_action_time > '2024-01-16'` + + You can also see section below [How to monitor tracking jobs](#how-to-monitor-tracking-jobs). + +3. incomplete graph might also come from archive processing. + + See [How to monitor archive jobs](#how-to-monitor-archive-jobs). + + +### How to monitor tracking jobs + +When a request arrive, it is logged into redis. +We then have matomo-tracking systemd services to process the redis queue. + +To see status: +```bash +# see jobs status +systemctl status matomo-tracking@{1..3}.service +# see timers status +systemctl status matomo-tracking@{1..3}.timer +``` + +Matomo also offers some commands to follow the status of the tracking jobs: + +```bash +cd /var/www/html/matomo +./console queuedtracking:monitor + +# you might also look at lock status +./console queuedtracking:lock-status +``` +use `--help` for more options. + +Logs are in `/var/log/matomo` + +### How to monitor archive jobs + +To see status: +```bash +# see jobs status +systemctl status matomo-archive.service +# see timers status +systemctl status matomo-archive.timer +``` + +Logs are in `/var/log/matomo` + +### How to monitor database + +`diagnostics:analyze-archive-table` can help see informations about tables. + +Use it with `YYYY_MM` argument. + +Eg. +```bash +cd /var/www/html/matomo +./console diagnostics:analyze-archive-table 2023_12 +``` + +`database:optimize-archive-tables` can optimize tables if needed. Do it on old tables that are not moving any more. Use with a `YYYY-MM-DD` argument. + +```bash +./console database:optimize-archive-table 2023-12-01 +``` + ### How to test a command in php cli For example I wanted to determine if we support async in CliMulti (used by `core::archive`). @@ -110,3 +199,4 @@ php > $p = new Piwik\CliMulti(); php > echo $p->supportsAsync(); 1 ``` + diff --git a/docs/munin.md b/docs/munin.md index 733a9ebc..22c61bae 100644 --- a/docs/munin.md +++ b/docs/munin.md @@ -34,10 +34,10 @@ To get zfs plugins we have to install contrib plugins. Clone In `/opt`: ```bash sudo git clone git@github.com:munin-monitoring/contrib.git munin-contrib -``` -sudo mkdir -p /usr/local/munin/lib/plugins +sudo mkdir -p /usr/local/munin/lib/ sudo ln -s /opt/munin-contrib/plugins /usr/local/munin/lib/ +``` #### ZFS plugins diff --git a/docs/reports/2024-01-matomo-perf-tunning b/docs/reports/2024-01-matomo-perf-tunning new file mode 100644 index 00000000..ffd58df0 --- /dev/null +++ b/docs/reports/2024-01-matomo-perf-tunning @@ -0,0 +1,230 @@ +# 2024-01 Matomo performance tunning + +We still have Matomo archive scripts failing… + +Trying to optimize it. + +**NOTE:** some of the changes documented here are in [commit 32c01fe796a1](https://github.com/openfoodfacts/openfoodfacts-infrastructure/commit/32c01fe796a119b377d19e8bf14cd81534ce9795) + +## using mysqltunner + +In a screen as root: `mysqltuner`, it is advized to let it run for 48h + +## PHP settings + +Edited `/etc/php/7.3/fpm/php.ini` and set as seen on /etc/php/7.3/fpm/php.ini + +(I only had to change max_execution_time) +```ini +memory_limit = 2G +max_execution_time = 0 +log_errors = On +display_errors=Off +``` + +For `/etc/php/7.3/cli/php.ini` +```ini +memory_limit = -1 +max_execution_time = 0 +log_errors = On +display_errors=Off +``` + +I also add the `path/to/matomo/config/common.config.ini.php` as advised (but with a different log path) + +Then `systemctl restart php7.3-fpm.service` + +## More MariaDB Optimization + +Followed [article on tmpfs for mysql on 2bits.com](https://2bits.com/articles/reduce-your-servers-resource-usage-moving-mysql-temporary-directory-ram-disk.html) (cited by [Matomo docs](https://matomo.org/faq/on-premise/how-to-configure-matomo-for-speed/)) to add: + +```conf +tmpdir=/run/mysqld +innodb_flush_log_at_trx_commit=2 +``` + +We could try to set `innodb_flush_method` to `O_DSYNC` but installed mariadb version does not support it. + +in `/etc/mysql/mariadb.conf.d/90-off-configs.cnf` + +Then `systemctl restart mariadb.service` + +Following https://mariadb.com/docs/server/ref/mdb/status-variables/Threads_created/ + +I run: + +```bash +MariaDB [(none)]> SELECT threads_created, connections, (threads_created / connections) AS thread_cache_miss_rate FROM (SELECT gs1.VARIABLE_VALUE AS threads_created FROM informat +ion_schema.GLOBAL_STATUS gs1 WHERE gs1.VARIABLE_NAME LIKE 'Threads_created') tc JOIN (SELECT gs2.VARIABLE_VALUE AS connections FROM information_schema.GLOBAL_STATUS gs2 WHERE + gs2.VARIABLE_NAME LIKE 'Connections') c; ++-----------------+-------------+------------------------+ +| threads_created | connections | thread_cache_miss_rate | ++-----------------+-------------+------------------------+ +| 9 | 239 | 0.03765690376569038 | ++-----------------+-------------+------------------------+ +1 row in set (0.001 sec) +``` + +thread_cache_miss_rate is almost 0 which is a good sign. + + +## Optimize the archiver launch + +Changed the exit time for `matomo-archive.service` to `5h` + +I use `/usr/bin/php /var/www/html/matomo/console core:archive --help` to see options. + +We could use `--force-idsites` to try to launch websites one by one, but I'm not sure it would help, for now (appart from launching archivers in parallel, but it may not help because bottleneck is certainly MariaDB). + + +## Trying to see what's in redis + + +Exploring with redis: +```bash +root@analytics:/var/log/matomo# redis-cli +127.0.0.1:6379> select 0 +OK +127.0.0.1:6379> keys * +1) "trackingQueueV1" +2) "QueuedTrackingLock0" +3) "fooList" +127.0.0.1:6379> llen trackingQueueV1 +(integer) 3886031 +127.0.0.1:6379> get QueuedTrackingLock0 +"f8e0ba116e18" +(...) +127.0.0.1:6379> get QueuedTrackingLock0 +"340aa64aebd3" +127.0.0.1:6379> ttl QueuedTrackingLock0 +(integer) 1910 +``` + +So we have a trackingQueueV1 with a lot of tracking records to process. (so it seems we didn't loose anything…) +Also we have a have a lock with a uuid, which changes after some time, and has a quite high ttl. + +## Some useful commands + +Reading at the [source code](https://github.com/matomo-org/plugin-QueuedTracking), +I also discovered some commands: + +There is a lock-status command +```bash +/usr/bin/php /var/www/html/matomo/console queuedtracking:lock-status --help +Usage: + queuedtracking:lock-status [--unlock="..."] + +Options: + --unlock If set will unlock the given queue. + --help (-h) Display this help message + --quiet (-q) Do not output any message + --verbose (-v|vv|vvv) Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug + --version (-V) Display this application version + --ansi Force ANSI output + --no-ansi Disable ANSI output + --no-interaction (-n) Do not ask any interactive question + --matomo-domain Matomo URL (protocol and domain) eg. "http://matomo.example.org" + --xhprof Enable profiling with XHProf + --ignore-warn Return 0 exit code even if there are warning logs or error logs detected in the command output. +``` + +and a monitoring command: +```bash +/usr/bin/php /var/www/html/matomo/console queuedtracking:monitor --help +Usage: + queuedtracking:monitor [--iterations="..."] + +Options: + --iterations If set, will limit the number of monitoring iterations done. + --help (-h) Display this help message + --quiet (-q) Do not output any message + --verbose (-v|vv|vvv) Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug + --version (-V) Display this application version + --ansi Force ANSI output + --no-ansi Disable ANSI output + --no-interaction (-n) Do not ask any interactive question + --matomo-domain Matomo URL (protocol and domain) eg. "http://matomo.example.org" + --xhprof Enable profiling with XHProf + --ignore-warn Return 0 exit code even if there are warning logs or error logs detected in the command output. +``` + +## Trying to launch the service by hand + +Stop current and verify it's not running +```bash +systemctl stop matomo-tracking.timer +systemctl status matomo-tracking.service +... dead +``` + +Start manually in a screen: +```bash +/usr/bin/php /var/www/html/matomo/console queuedtracking:process -v +``` + +It seems to be processing. To verify let's look at the database. + +When we started, in mysql we had: +```SQL +MariaDB [matomo_db]> select count(*) from matomo_log_visit where visit_last_action_time > '2024-01-1 +6'; ++----------+ +| count(*) | ++----------+ +| 44581 | ++----------+ +`````` + +1 h after: +```SQL +MariaDB [matomo_db]> select count(*) from matomo_log_visit where visit_last_action_time > '2024-01-16'; ++----------+ +| count(*) | ++----------+ +| 47785 | ++----------+ +``` + +So it's working but it's quite slow (~3000 / h). + +## Augmenting the service max execution time + +The problem with current service settings is that the TimeoutStartSec is maybe too low. +When a tracking service is stopped, it might leave the lock behind it, with a ttl that might be a bit high, so it then restart a lot of time without doing anything because of the lock. + +## Going to 4 tracker queues + +Matomo propose to use more than one queue to handle incoming requests. This is what we will do. + +Modification of [matomo Queuetracker plugin settings](https://analytics.openfoodfacts.org/index.php?module=CoreAdminHome&action=generalSettings&idSite=1&period=day&date=yesterday&activated=#/QueuedTracking) to have 4 queues. + +Modification of matomo-tracking service to run 4 services and queues (using instance name). + +```bash +ln -s /opt/openfoodfacts-infrastructure/confs/matomo/systemd/matomo-tracking@.service /etc/systemd/system +ln -s /opt/openfoodfacts-infrastructure/confs/matomo/systemd/matomo-tracking@.timer /etc/systemd/system +systemctl disable matomo-tracking.timer +unlink /etc/systemd/system/matomo-tracking.service +unlink /etc/systemd/system/matomo-tracking.timer +systemctl daemon-reload +systemctl enable matomo-tracking@{0,1,2,3}.timer +systemctl start matomo-tracking@{0,1,2,3}.timer +``` + +See [commit 640895428](https://github.com/openfoodfacts/openfoodfacts-infrastructure/commit/640895428351826b22f571327de3df2ee6f6e76c) + +We can see it's running: +```bash +systemctl status matomo-tracking@{0..3}.timer +``` + +In redis we see the different queues that have been created: +```bash +$ redis-cli +127.0.0.1:6379> keys * +1) "QueuedTrackingLock0" +3) "trackingQueueV1_2" +4) "trackingQueueV1" +5) "trackingQueueV1_3" +6) "fooList" +```