tweets = ['this tweet has a link htt://link','this one does not','this one does http://link.net']
我使用sklearn得到了文本数据的稀疏矩阵。
tfidf_vectorizer = TfidfVectorizer(max_df=0.90, max_features=200000,
min_df=0.1, stop_words='english',
use_idf=True, tokenizer=nltk.tokenize, ngram_range=(1,2))
tfidf_matrix = tfidf_vectorizer.fit_transform(tweets)
我希望为其添加列以支持文本数据的其他特征。我尝试了以下方法:
import scipy as sc
all_data = sc.hstack((tfidf_matrix, [1,0,1]))
array([ <3x8 sparse matrix of type '<type 'numpy.float64'>'
with 10 stored elements in Compressed Sparse Row format>,
1, 1, 0], dtype=object)
`from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB().fit(all_data, y)`
我遇到了一个回溯错误:
`Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Anaconda\lib\site- packages\spyderlib\widgets\externalshell\sitecustomize.py", line 580, in runfile
execfile(filename, namespace)
File "C:/Users/c/Desktop/features.py", line 157, in <module>
clf = MultinomialNB().fit(all_data, y)
File "C:\Anaconda\lib\site-packages\sklearn\naive_bayes.py", line 302, in fit
_, n_features = X.shape
数值错误:需要至少1个值来解包
编辑:数据的形状
`tfidf_matrix.shape
(100, 2)
all_data.shape
(100L,)`
我可以直接将列添加到稀疏矩阵中吗?如果不行,我该如何将数据转换为支持此操作的格式?我担心使用稀疏矩阵以外的其他格式会增加内存占用。
MultinomialNB().fit(tfidf_matrix, y)
吗? - Alex Plugarutfidf_matrix
和all_data
的shape
可能不匹配?也许你应该这样做:根据这里的解释,all_data = sc.hstack((tfidf_matrix, [[1],[0],[1]]))
:http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.hstack.html - Alex Plugaruall_data.reshape(tfidf_matrix.shape)
。 - Alex Plugaru