Neo4j嵌入式Java - 事务被标记为成功，但无法提交事务，因此被回滚。

Question

我正在试图在Neo4j中加载一个包含17,210,368个关系的16807个节点的图形。为此，我会加载一个包含邻近表的文件，并获得一个必须连接到关系的节点列表。

以下是我的代码：

    String inputFile = "Desktop\kron7cd_unix.t01";
    FileInputStream in = new FileInputStream(inputFile);
    FileChannel ch = in.getChannel();
    ByteBuffer buf = ByteBuffer.allocate(1024);

    ArrayList<Integer> list = new ArrayList<Integer>();
    int NumOfOnes = 0;
    int column=-1;
    int row=0;
    int rd;
    while ((rd = ch.read( buf )) != -1){
        buf.flip();
        while (buf.hasRemaining()){
            byte byteVal = buf.get();
            if((byteVal == 48) || (byteVal == 49)){// when finds 1 or 0
                column++;
            }
            if (byteVal == 92){//when finds '/'
                    row++;
                    column=-1;
            }
            if(byteVal == 49){//when finds 1
                NumOfOnes++;
                list.add(column);
                list.add(row);
            }
        }
    buf.clear();
    }
    ch.close();


    GraphDatabaseFactory dbFactory = new GraphDatabaseFactory();
    GraphDatabaseService graphDb = dbFactory.newEmbeddedDatabase("C:\Neo4j\default.graphdb");

    Transaction tx = graphDb.beginTx();
    try {

        Label myLabel = DynamicLabel.label("Data");
        ArrayList<Node> nodelist = new ArrayList<Node>();

        for (int k = 0; k < row; k++) {
            nodelist.add(graphDb.createNode());
        }

        for (int k = 0; k < row; k++) {
            nodelist.get(k).setProperty("ID", k);
            nodelist.get(k).setProperty("Group","Random");
            nodelist.get(k).addLabel(myLabel);
        }

        Relationship rel;
        final RelationshipType type2 = DynamicRelationshipType.withName("Rel");

        for (int j = 0; j < list.size()-1 ; j += 2) { //list.size()=34420736

            rel = nodelist.get(list.get(j)).createRelationshipTo(nodelist.get(list.get(j+1)), type2);
            rel.setProperty("Team", "Common");

            if (j > 0 && j % 10000 == 0) {// as to commit transaction every now and then and dont throw heap space
                tx.success();
                tx.close();
                tx = graphDb.beginTx();
            }                

        }

        tx.success();
    }

    finally {
        tx.close();
    }
    graphDb.shutdown();

当我运行这段代码时，它抛出以下错误。我使用的是Neo4j 2.3.3和带有Java 8的Netbeans 8.1。我想知道问题是堆空间还是在尝试提交事务时出现的。我还将命令行选项-Xmx1g添加到我的项目中，以增加堆空间。

有任何想法吗？

下面是错误消息：

Exception in thread "main" org.neo4j.graphdb.TransactionFailureException: Transaction was marked as successful, but unable to commit transaction so rolled back.
at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:121)
at com.mycompany.traverse_test.traverse_main.main(traverse_main.java:232)

Caused by: org.neo4j.kernel.api.exceptions.TransactionFailureException: Could not apply the transaction to the store after written to log
at org.neo4j.kernel.impl.api.TransactionRepresentationCommitProcess.applyToStore(TransactionRepresentationCommitProcess.java:105)
at org.neo4j.kernel.impl.api.TransactionRepresentationCommitProcess.commit(TransactionRepresentationCommitProcess.java:58)
at org.neo4j.kernel.impl.api.KernelTransactionImplementation.commit(KernelTransactionImplementation.java:565)
at org.neo4j.kernel.impl.api.KernelTransactionImplementation.close(KernelTransactionImplementation.java:458)
at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:97)
... 1 more

Caused by: java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:386)
at org.neo4j.unsafe.impl.internal.dragons.MemoryManager$Slab.<init>(MemoryManager.java:111)
at org.neo4j.unsafe.impl.internal.dragons.MemoryManager.allocateAligned(MemoryManager.java:82)
at org.neo4j.io.pagecache.impl.muninn.MuninnPage.initBuffer(MuninnPage.java:417)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:230)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:157)
at org.neo4j.io.pagecache.impl.muninn.MuninnWritePageCursor.next(MuninnWritePageCursor.java:58)
at org.neo4j.kernel.impl.store.PropertyStore.updateRecord(PropertyStore.java:144)
at org.neo4j.kernel.impl.transaction.command.NeoStoreTransactionApplier.visitPropertyCommand(NeoStoreTransactionApplier.java:99)
at org.neo4j.kernel.impl.api.CommandApplierFacade.visitPropertyCommand(CommandApplierFacade.java:120)
at org.neo4j.kernel.impl.transaction.command.Command$PropertyCommand.handle(Command.java:288)
at org.neo4j.kernel.impl.api.CommandApplierFacade.visit(CommandApplierFacade.java:82)
at org.neo4j.kernel.impl.api.CommandApplierFacade.visit(CommandApplierFacade.java:45)
at org.neo4j.kernel.impl.transaction.log.PhysicalTransactionRepresentation.accept(PhysicalTransactionRepresentation.java:69)
at org.neo4j.kernel.impl.api.TransactionRepresentationStoreApplier.apply(TransactionRepresentationStoreApplier.java:111)
at org.neo4j.kernel.impl.api.TransactionRepresentationCommitProcess.applyToStore(TransactionRepresentationCommitProcess.java:100)
... 5 more

- dimcode

Answer 1

你应该每1k - 10k个元素提交一次事务，以避免将所有数据都保留在堆上。例如，请参阅https://github.com/graphaware/neo4j-framework/tree/master/tx-executor#batch-transactional-operations。

- Michal Bachman

---

我有点困惑。我认为我代码中的语句 if (j > 0 && j % 10000 == 0) 正是做了这件事情，即在创建了10k个关系后提交一次事务。 - dimcode

是的，抱歉。你能试着把它改成1000吗？如果这样还不行，可以考虑使用类似YourKit profiler的工具来查看实际占用内存的情况。 - Michal Bachman

Answer 2

通过在另一个系统中尝试运行相同的代码，我意识到问题是堆空间错误。我在我的系统上运行代码时有3GB RAM，它弹出上述错误，但当我在具有12GB RAM的系统上运行它时，它正常运行。（我想即使有8GB RAM也不会有问题）

- dimcode