如果您要使用WordNet,您会遇到以下问题:
问题1:词义消歧(WSD),即如何自动确定使用哪个同义词集?
>>> for i in wn.synsets('good','a'):
... print i.name, i.definition
...
good.a.01 having desirable or positive qualities especially those suitable for a thing specified
full.s.06 having the normally expected amount
good.a.03 morally admirable
estimable.s.02 deserving of esteem and respect
beneficial.s.01 promoting or enhancing well-being
good.s.06 agreeable or pleasing
good.s.07 of moral excellence
adept.s.01 having or showing knowledge and skill and aptitude
good.s.09 thorough
dear.s.02 with or in a close or intimate relationship
dependable.s.04 financially sound
good.s.12 most suitable or right for a particular purpose
good.s.13 resulting favorably
effective.s.04 exerting force or influence
good.s.15 capable of pleasing
good.s.16 appealing to the mind
good.s.17 in excellent physical condition
good.s.18 tending to promote physical well-being; beneficial to health
good.s.19 not forged
good.s.20 not left to spoil
good.s.21 generally admired
>>> for i in wn.synsets('great','a'):
... print i.name, i.definition
...
great.s.01 relatively large in size or number or extent; larger than others of its kind
great.s.02 of major significance or importance
great.s.03 remarkable or out of the ordinary in degree or magnitude or effect
bang-up.s.01 very good
capital.s.03 uppercase
big.s.13 in an advanced stage of pregnancy
假设你已经以某种方式得出了正确的意思,也许你尝试过像这样(
https://github.com/alvations/pywsd),并且假设你成功获取了词性和同义词集:
good.a.01 具有令人满意或积极的特质,特别是适用于指定事物的特质。
great.s.01 相对较大,数量或范围较大;比同类中的其他对象更大。
问题2: 如何比较这两个同义词集?
让我们尝试相似度函数,但你会发现它们没有给你任何分数:
>>> good = wn.synsets('good','a')[0]
>>> great = wn.synsets('great','a')[0]
>>> print max(wn.path_similarity(good,great), wn.path_similarity(great, good))
None
>>> print max(wn.wup_similarity(good,great), wn.wup_similarity(great, good))
>>> print max(wn.res_similarity(good,great,semcor_ic), wn.res_similarity(great, good,semcor_ic))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/nltk/corpus/reader/wordnet.py", line 1312, in res_similarity
return synset1.res_similarity(synset2, ic, verbose)
File "/usr/local/lib/python2.7/dist-packages/nltk/corpus/reader/wordnet.py", line 738, in res_similarity
ic1, ic2, lcs_ic = _lcs_ic(self, other, ic)
File "/usr/local/lib/python2.7/dist-packages/nltk/corpus/reader/wordnet.py", line 1643, in _lcs_ic
(synset1, synset2))
nltk.corpus.reader.wordnet.WordNetError: Computing the least common subsumer requires Synset('good.a.01') and Synset('great.s.01') to have the same part of speech.
>>> print max(wn.jcn_similarity(good,great,semcor_ic), wn.jcn_similarity(great, good,semcor_ic))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/nltk/corpus/reader/wordnet.py", line 1316, in jcn_similarity
return synset1.jcn_similarity(synset2, ic, verbose)
File "/usr/local/lib/python2.7/dist-packages/nltk/corpus/reader/wordnet.py", line 759, in jcn_similarity
ic1, ic2, lcs_ic = _lcs_ic(self, other, ic)
File "/usr/local/lib/python2.7/dist-packages/nltk/corpus/reader/wordnet.py", line 1643, in _lcs_ic
(synset1, synset2))
nltk.corpus.reader.wordnet.WordNetError: Computing the least common subsumer requires Synset('good.a.01') and Synset('great.s.01') to have the same part of speech.
>>> print max(wn.lin_similarity(good,great,semcor_ic), wn.lin_similarity(great, good,semcor_ic))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/nltk/corpus/reader/wordnet.py", line 1320, in lin_similarity
return synset1.lin_similarity(synset2, ic, verbose)
File "/usr/local/lib/python2.7/dist-packages/nltk/corpus/reader/wordnet.py", line 789, in lin_similarity
ic1, ic2, lcs_ic = _lcs_ic(self, other, ic)
File "/usr/local/lib/python2.7/dist-packages/nltk/corpus/reader/wordnet.py", line 1643, in _lcs_ic
(synset1, synset2))
nltk.corpus.reader.wordnet.WordNetError: Computing the least common subsumer requires Synset('good.a.01') and Synset('great.s.01') to have the same part of speech.
>>> print max(wn.lch_similarity(good,great), wn.lch_similarity(great, good))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/nltk/corpus/reader/wordnet.py", line 1304, in lch_similarity
return synset1.lch_similarity(synset2, verbose, simulate_root)
File "/usr/local/lib/python2.7/dist-packages/nltk/corpus/reader/wordnet.py", line 638, in lch_similarity
(self, other))
nltk.corpus.reader.wordnet.WordNetError: Computing the lch similarity requires Synset('good.a.01') and Synset('great.s.01') to have the same part of speech.
让我们尝试不同的同义词集合,因为“good”既有“卫星形容词”,也有“形容词”,而“great”只有“卫星”,让我们选择最低共同分母:
good.s.13 resulting favorably
great.s.01 relatively large in size or number or extent; larger than others of its kind
您意识到仍然没有相似性信息可用于比较“卫星形容词”之间的差异。
>>> print max(wn.lin_similarity(good,great,semcor_ic), wn.lin_similarity(great, good,semcor_ic))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/nltk/corpus/reader/wordnet.py", line 1320, in lin_similarity
return synset1.lin_similarity(synset2, ic, verbose)
File "/usr/local/lib/python2.7/dist-packages/nltk/corpus/reader/wordnet.py", line 789, in lin_similarity
ic1, ic2, lcs_ic = _lcs_ic(self, other, ic)
File "/usr/local/lib/python2.7/dist-packages/nltk/corpus/reader/wordnet.py", line 1645, in _lcs_ic
ic1 = information_content(synset1, ic)
File "/usr/local/lib/python2.7/dist-packages/nltk/corpus/reader/wordnet.py", line 1666, in information_content
raise WordNetError(msg % synset.pos)
nltk.corpus.reader.wordnet.WordNetError: Information content file has no entries for part-of-speech: s
>>> print max(wn.path_similarity(good,great), wn.path_similarity(great, good))None
None
现在似乎WordNet正在创造更多问题,而不是解决任何问题。让我们尝试另一种方法,尝试单词聚类,请参见链接:
http://en.wikipedia.org/wiki/Word-sense_induction
我也放弃回答OP提出的广泛和开放的问题,因为在聚类中有很多自动化的东西对于像我这样的平凡人来说太复杂了 =)
['优质服务','良好产品','实惠价格']
,主题2-['良好服务','高品质产品','合理价格']
。没有人会将它们标记为两个不同的主题,而且当存在其他可能被视为主题的内容时,这并不实用。良好产品
和优质产品
都以积极的方式描述产品,您可以从易于参考
的角度看出它们更相似。 - user2979931