Of interest for the data science angle. Also it includes the word “bayesian” and links to Paul Graham. That’s got to make it more or less on-topic, right?
This was mostly just a fun exercise. I wasn’t expecting to produce a usable product (and I didn’t), but as an exercise it was pretty informative and well worth my time. I’ve put some observations in rot13, as anyone else undertaking this exercise may benefit from observing them by themselves.
Vg’f n irel tbbq qrzbafgengvba bs cbjre nanylfvf pbaprcgf, naq gur genqr-bss orgjrra frafvgvivgl naq fcrpvsvpvgl. Jura znxvat gjrnxf gb gur pynffvsvre, vg ernyyl unzzref ubzr gur vqrn gung ryvzvangvat glcr-V reebef pneevrf gur pbfg bs zber glcr-VV reebef.
Gur birenyy zrgubq (anvir onlrf pynffvsvpngvba) frrzf gb or fvtavsvpnagyl yrff hfrshy sbe grkg zrffntrf guna sbe rznvy zrffntrf. Gurer ner n ahzore bs boivbhf ernfbaf sbe guvf, naq n srj yrff boivbhf barf. Grkg zrffntrf ner n ybg fubegre guna rznvyf (hfhnyyl yrff guna 140 punenpgref, juvyr rznvyf jvyy abeznyyl unir ng yrnfg 140 punenpgref bs urnqre zrgnqngn orsber lbh rira trg bagb gur obql grkg). Nyfb, yrtvgvzngr grkg zrffntrf frrz gb pbagnva n ybg zber “fcnzzl” pbagrag guna rznvyf. Gur zrffntrf va zl grfg unz pbechf gung synttrq nf fcnz nyy ybbxrq ernyyl fcnzzl. Gur zbfg ragregnvavat qvfpbirel jnf gung gur gbxra “kkk” ybbxrq zber yrtvg guna fcnzzl orpnhfr fb znal crbcyr fvta bss gurve grkgf jvgu gung.
Of interest for the data science angle. Also it includes the word “bayesian” and links to Paul Graham. That’s got to make it more or less on-topic, right?
I recently took this collection of text messages (from Hilary Mason’s collection of research-quality data sets ), and attempted to build a simple Bayesian spam filter for text messages.
This was mostly just a fun exercise. I wasn’t expecting to produce a usable product (and I didn’t), but as an exercise it was pretty informative and well worth my time. I’ve put some observations in rot13, as anyone else undertaking this exercise may benefit from observing them by themselves.
Vg’f n irel tbbq qrzbafgengvba bs cbjre nanylfvf pbaprcgf, naq gur genqr-bss orgjrra frafvgvivgl naq fcrpvsvpvgl. Jura znxvat gjrnxf gb gur pynffvsvre, vg ernyyl unzzref ubzr gur vqrn gung ryvzvangvat glcr-V reebef pneevrf gur pbfg bs zber glcr-VV reebef.
Gur birenyy zrgubq (anvir onlrf pynffvsvpngvba) frrzf gb or fvtavsvpnagyl yrff hfrshy sbe grkg zrffntrf guna sbe rznvy zrffntrf. Gurer ner n ahzore bs boivbhf ernfbaf sbe guvf, naq n srj yrff boivbhf barf. Grkg zrffntrf ner n ybg fubegre guna rznvyf (hfhnyyl yrff guna 140 punenpgref, juvyr rznvyf jvyy abeznyyl unir ng yrnfg 140 punenpgref bs urnqre zrgnqngn orsber lbh rira trg bagb gur obql grkg). Nyfb, yrtvgvzngr grkg zrffntrf frrz gb pbagnva n ybg zber “fcnzzl” pbagrag guna rznvyf. Gur zrffntrf va zl grfg unz pbechf gung synttrq nf fcnz nyy ybbxrq ernyyl fcnzzl. Gur zbfg ragregnvavat qvfpbirel jnf gung gur gbxra “kkk” ybbxrq zber yrtvg guna fcnzzl orpnhfr fb znal crbcyr fvta bss gurve grkgf jvgu gung.
Zl vaghvgvbaf nobhg ubj cebonovyvgvrf hcqngr tvira bgure cebonovyvgvrf vf nyy jebat.
If anyone else tries this, let me know how you get on.
This is great, thanks. I’ll implement a rudimentary spam filter with this sometime next week.
What is the maximum F1 score you can obtain?