logo标签列表

Elasticsearch - Java RestHighLevelClient - 如何使用滚动API获取所有文档

javaelasticsearch
3

在我的Elasticsearch索引中，我保存了约30000个实体。我想使用RestHighLevelClient获取它们的所有ID。我已经阅读到最好的方法是使用滚动API来完成。但是当我这样做时，我只收到大约10个实体而不是30k。如何解决这个问题？

final class ElasticRepo {
    private final RestHighLevelClient restHighLevelClient;

List<ListingsData> getAllListingsDataIds() {
        val request = new SearchRequest(ELASTICSEARCH_LISTINGS_INDEX);
        request.types(ELASTICSEARCH_TYPE);
        val searchSourceBuilder = new SearchSourceBuilder()
                .query(matchAllQuery())
                .fetchSource(new String[]{"listing_id"}, new String[]{"backoffice_data", "search_and_match_data"});
        request.source(searchSourceBuilder);
        request.scroll(TimeValue.timeValueMinutes(3));
        return executeQuery(request);
    }

 private List<ListingsData> executeQuery(final SearchRequest searchQuery) {
        try {
            val hits = restHighLevelClient.search(searchQuery, RequestOptions.DEFAULT).getHits().getHits();
            return Arrays.stream(hits).map(SearchHit::getSourceAsString).map(ElasticRepo::toListingsData).collect(Collectors.toList());
        } catch (IOException e) {
            e.printStackTrace();
            throw new RuntimeException("");
        }
    }

}

当我执行executeQuery时，它只返回大约11个实体。如何解决这个问题，如何获取索引中的所有文档？
如果您发现这些答案中有一个有用，请接受其中之一。 - Aymen Ragoubi
好的，我还没有测试过它。 - Clyde Barrow
2个回答
10

尝试按照这个例子进行，我正在使用这段代码，它可以正常工作：

        String query = "your query here";

        QueryBuilder matchQueryBuilder = QueryBuilders.boolQuery().must(new QueryStringQueryBuilder(query));

        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

        searchSourceBuilder.query(matchQueryBuilder);

        searchSourceBuilder.size(5000); //max is 10000

        searchRequest.indices("your index here");

        searchRequest.source(searchSourceBuilder);

        final Scroll scroll = new Scroll(TimeValue.timeValueMinutes(10L));

        searchRequest.scroll(scroll);

        SearchResponse searchResponse = client.search(searchRequest);
            String scrollId = searchResponse.getScrollId();

        SearchHit[] allHits = new SearchHit[0];

        SearchHit[] searchHits = searchResponse.getHits().getHits();

        while (searchHits != null && searchHits.length > 0)
        {

            allHits = Helper.concatenate(allHits, searchResponse.getHits().getHits()); //create a function which concatenate two arrays

            SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);

            scrollRequest.scroll(scroll);

            searchResponse = client.searchScroll(scrollRequest);

            scrollId = searchResponse.getScrollId();

            searchHits = searchResponse.getHits().getHits();

        }

        ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
        clearScrollRequest.addScrollId(scrollId);
        ClearScrollResponse clearScrollResponse = client.clearScroll(clearScrollRequest);
2
网页内容由stack overflow 提供, 点击上面的
可以查看英文原文，
原文链接