logo标签列表

使用自然语言处理进行句子压缩

nlpnltkstanford-nlpopennlp
16

使用机器翻译，我可以获得一个非常压缩的句子版本，例如我真的很想喝一杯美味可口的咖啡将被翻译为我想喝咖啡。是否有任何NLP引擎提供这样的功能？

我找到了一些进行释义生成和句子压缩的研究论文。但是是否有已经实现了这个功能的库？

3
我不知道有哪个工具可以做到这一点，但是进行解析，然后删除形容词短语中的副词和一些其他结构可能会给您提供一个不错的基准。 - Fred Foo
2
你可以删掉形容词/副词，但是你在上面的例子中所指示的是压缩动词形式，即“would really like to have” -> “want”。此外，“tasty cup of coffee”变成“coffee”？有很多情况下你想要获取根名词，比如说“镇上的汽车经销商”。我不知道有什么工具可以做到这一点。 - nflacco
我也会在metaoptimize.com/qa/上发布帖子。您可以尝试联系James Clarke，网址为http://jamesclarke.net。 - cyborg
我不禁想起《迷失东京》中的Suntory广告：“面对镜头...带着强烈的情感。” http://www.youtube.com/watch?v=FiQnH450hPM - Iterator
根据我们的主题相关指南，“即使一些问题符合上述类别，仍然有些问题是不适合讨论的： ……要求我们推荐或查找书籍、工具、软件库、教程或其他场外资源的问题是不适合讨论的……” - Robert Columbia
4个回答
11

如果你想使你的句子简短而不失去重要思想，那么你可以通过提取三元组主语-谓语-宾语来实现。

谈到工具/引擎，我推荐使用斯坦福自然语言处理器。它的依赖解析器输出已经提供了主语和宾语（如果有）。但是您仍需要进行一些调整以获得所需的结果。

您可以在这里下载斯坦福自然语言处理器并学习样例用法。

我发现与您问题相关的论文。请查看Text Simplification using Typed Dependencies: A Comparison of the Robustness of Different Generation Strategie

2

以下是我发现的内容：

这是一个修改后的模型实现，描述在Clarke和Lapata，2008年“句子压缩的全局推理：整数线性规划方法”中。

论文：https://www.jair.org/media/2433/live-2433-3731-jair.pdf

源代码：https://github.com/cnap/sentence-compression（使用JAVA编写）

输入：在营地里，反叛军队被一面写着“欢迎回家”的横幅所欢迎。

输出：在营地里，士兵们受到了欢迎。

更新： 用于文本摘要的序列到序列注意力模型。

https://github.com/tensorflow/models/tree/master/textsum

https://arxiv.org/abs/1509.00685

1
首先尝试使用Watson NaturalLanguageUnderstanding/Alchemy库。使用它，我能够从我的陈述中提取重要关键词，例如：
输入：嘿！我的笔记本电脑屏幕出现问题。 输出：笔记本电脑屏幕问题硬件。
不仅可以重新表述，而且使用NLU可以获取输入语句的以下细节，例如对于上述语句，您可以获取以下类别的详细信息：
语言，如“en”，实体，概念，关键字如“笔记本电脑屏幕”，“问题”及其相关性、文本、关键字情感、情绪。 带有标签和相关性分数的类别。 带有详细信息的SemanticRoles，例如句子、主语、动作和对象。
除此之外，您还可以使用语气分析器来获取陈述的显著语气，例如恐惧、愤怒、快乐、厌恶等。
以下是Watson库的代码示例 注意：Watson库不免费，但提供一个月的试用期，因此您可以从这里开始，一旦掌握了概念，就可以切换到其他开源库，并找出类似的库和函数。
NaturalLanguageUnderstanding service = new NaturalLanguageUnderstanding(
    NaturalLanguageUnderstanding.VERSION_DATE_2017_02_27,
    WatsonConfiguration.getAlchemyUserName(),
    WatsonConfiguration.getAlchemyPassword());


//ConceptsOptions
ConceptsOptions conceptOptions = new ConceptsOptions.Builder()
    .limit(10)
    .build();

//CategoriesOptions
    CategoriesOptions categoriesOptions = new CategoriesOptions();

//SemanticOptions
SemanticRolesOptions semanticRoleOptions = new SemanticRolesOptions.Builder()
    .entities(true)
    .keywords(true)
    .limit(10)
    .build();

EntitiesOptions entitiesOptions = new EntitiesOptions.Builder()
    .emotion(true)
    .sentiment(true)
    .limit(10)
    .build();

KeywordsOptions keywordsOptions = new KeywordsOptions.Builder()
    .emotion(true)
    .sentiment(true)
    .limit(10)
    .build();

Features features = new Features.Builder()
    .entities(entitiesOptions)
    .keywords(keywordsOptions)
    .concepts(conceptOptions)
    .categories(categoriesOptions)
    .semanticRoles(semanticRoleOptions)
    .build();

AnalyzeOptions parameters = new AnalyzeOptions.Builder()
    .text(inputText)
    .features(features)
    .build();

AnalysisResults response = service
    .analyze(parameters)
    .execute();
System.out.println(response);
0
你可以结合“停用词去除”和“词干提取与词形还原”来进行编程处理。 词干提取与词形还原是将文本中的所有单词返回到其基本根源的过程，你可以在这里找到完整的解释。我正在使用Porter词干提取器，在谷歌上查找一下。 在进行了词干提取和词形还原之后，停用词的去除就非常容易了，这是我的停用词去除方法：
public static String[] stopwords ={"a", "about", "above", "across", "after", "afterwards", "again", "against", "all", "almost", 
    "alone", "along", "already", "also","although","always","am","among", "amongst", "amoungst", "amount",  "an", "and", 
    "another", "any","anyhow","anyone","anything","anyway", "anywhere", "are", "around", "as",  "at", "back","be","became", 
    "because","become","becomes", "becoming", "been", "before", "beforehand", "behind", "being", "below", "beside", "besides", 
    "between", "beyond", "bill", "both", "bottom","but", "by", "call", "can", "cannot", "cant", "co", "con", "could", "couldnt",
    "cry", "de", "describe", "detail", "do", "done", "down", "due", "during", "each", "eg", "eight", "either", "eleven","else",
    "elsewhere", "empty", "enough", "etc", "even", "ever", "every", "everyone", "everything", "everywhere", "except", "few", 
    "fifteen", "fify", "fill", "find", "fire", "first", "five", "for", "former", "formerly", "forty", "found", "four", "from", 
    "front", "full", "further", "get", "give", "go", "had", "has", "hasnt",
    "have", "he", "hence", "her", "here", "hereafter", "hereby", "herein", "hereupon", "hers", "herself", 
    "him", "himself", "his", "how", "however", "hundred", "ie", "if", "in", "inc", "indeed", "interest", "into", 
    "is", "it", "its", "itself", "keep", "last", "latter", "latterly", "least", "less", "ltd", "made", "many", 
    "may", "me", "meanwhile", "might", "mill", "mine", "more", "moreover", "most", "mostly", "move", "much", "must", 
    "my", "myself", "name", "namely", "neither", "never", "nevertheless", "next", "nine", "no", "nobody", "none", 
    "noone", "nor", "not", "nothing", "now", "nowhere", "of", "off", "often", "on", "once", "one", "only", "onto", 
    "or", "other", "others", "otherwise", "our", "ours", "ourselves", "out", "over", "own","part", "per", "perhaps",
    "please", "put", "rather", "re", "same", "see", "seem", "seemed", "seeming", "seems", "serious", "several", "she",
    "should", "show", "side", "since", "sincere", "six", "sixty", "so", "some", "somehow", "someone", "something", 
    "sometime", "sometimes", "somewhere", "still", "such", "system", "take", "ten", "than", "that", "the", "their", 
    "them", "themselves", "then", "thence", "there", "thereafter", "thereby", "therefore", "therein", "thereupon", 
    "these", "they", "thickv", "thin", "third", "this", "those", "though", "three", "through", "throughout", "thru", 
    "thus", "to", "together", "too", "top", "toward", "towards", "twelve", "twenty", "two", "un", "under", "until", 
    "up", "upon", "us", "very", "via", "was", "we", "well", "were", "what", "whatever", "when", "whence", "whenever",
    "where", "whereafter", "whereas", "whereby", "wherein", "whereupon", "wherever", "whether", "which", "while", 
    "whither", "who", "whoever", "whole", "whom", "whose", "why", "will", "with", "within", "without", "would", "yet",
    "you", "your", "yours", "yourself", "yourselves","1","2","3","4","5","6","7","8","9","10","1.","2.","3.","4.","5.","6.","11",
    "7.","8.","9.","12","13","14","A","B","C","D","E","F","G","H","I","J","K","L","M","N","O","P","Q","R","S","T","U","V","W","X","Y","Z",
    "terms","CONDITIONS","conditions","values","interested.","care","sure","!","@","#","$","%","^","&","*","(",")","{","}","[","]",":",";",",","<",">","/","?","_","-","+","=",
    "a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z",
    "contact","grounds","buyers","tried","said,","plan","value","principle.","forces","sent:","is,","was","like",
    "discussion","tmus","diffrent.","layout","area.","thanks","thankyou","hello","bye","rise","fell","fall","psqft.","http://","km","miles"};

在我的项目中，我使用段落作为文本输入：
public static String removeStopWords(String paragraph) throws IOException{
    Scanner paragraph1=new Scanner( paragraph );
    String newtext="";
    Map map = new TreeMap();
    Integer ONE = new Integer(1);
    while(paragraph1.hasNext()) {
        int flag=1;
        fixString=paragraph1.next();
        fixString=fixString.toLowerCase();
        for(int i=0;i<stopwords.length;i++) {
            if(fixString.equals(stopwords[i])) {
                flag=0;
            }
        }
        if(flag!=0) {
            newtext=newtext+fixString+" ";  
        }
            if (fixString.length() > 0) {
            Integer frequency = (Integer) map.get(fixString);
            if (frequency == null) {
                frequency = ONE;
            } else {
                int value = frequency.intValue();
                frequency = new Integer(value + 1);
            }
            map.put(fixString, frequency);                 
            }                     
    }
    return newtext;
}

我使用了斯坦福自然语言处理库，你可以从这里下载。

希望我能在某种程度上帮到你。

链接暂时无法使用。此外，仅仅删除停用词是不够的，正如 @nflacco 所评论的那样。总体而言，这个答案应该得到改进。 - mins
我已经检查了链接，它们都能正常工作。我知道我的答案并不完美（在自然语言处理中没有什么是完美的），但这是一个开始。 - KhuzG
你会建议我在我的回答中改进什么？ - KhuzG
1
欢迎来到本站。您的回答是一个有趣的贡献，如果您修复一些方面，它将很可能得到更多投票：由于某种原因，第一个链接已失效。正如在如何撰写好的答案？中所解释的那样，提供链接的上下文是一个好习惯。这也会平衡对词干提取和词形还原与停用词去除步骤的重要性。 - mins
你应该定义代码的目的和用途，而不是仅仅发布代码，并以一种语言无关的方式提供解决方案。 - Ravinder Payal
1
我认为他需要进行一些语义分析来提取信息，但这只是预处理。 - Raymond Chen
网页内容由stack overflow 提供, 点击上面的
可以查看英文原文，
原文链接