我正在试图在Neo4j中加载一个包含17,210,368个关系的16807个节点的图形。为此,我会加载一个包含邻近表的文件,并获得一个必须连接到关系的节点列表。
以下是我的代码:
当我运行这段代码时,它抛出以下错误。我使用的是Neo4j 2.3.3和带有Java 8的Netbeans 8.1。我想知道问题是堆空间还是在尝试提交事务时出现的。我还将命令行选项-Xmx1g添加到我的项目中,以增加堆空间。
有任何想法吗?
下面是错误消息:
以下是我的代码:
String inputFile = "Desktop\kron7cd_unix.t01";
FileInputStream in = new FileInputStream(inputFile);
FileChannel ch = in.getChannel();
ByteBuffer buf = ByteBuffer.allocate(1024);
ArrayList<Integer> list = new ArrayList<Integer>();
int NumOfOnes = 0;
int column=-1;
int row=0;
int rd;
while ((rd = ch.read( buf )) != -1){
buf.flip();
while (buf.hasRemaining()){
byte byteVal = buf.get();
if((byteVal == 48) || (byteVal == 49)){// when finds 1 or 0
column++;
}
if (byteVal == 92){//when finds '/'
row++;
column=-1;
}
if(byteVal == 49){//when finds 1
NumOfOnes++;
list.add(column);
list.add(row);
}
}
buf.clear();
}
ch.close();
GraphDatabaseFactory dbFactory = new GraphDatabaseFactory();
GraphDatabaseService graphDb = dbFactory.newEmbeddedDatabase("C:\Neo4j\default.graphdb");
Transaction tx = graphDb.beginTx();
try {
Label myLabel = DynamicLabel.label("Data");
ArrayList<Node> nodelist = new ArrayList<Node>();
for (int k = 0; k < row; k++) {
nodelist.add(graphDb.createNode());
}
for (int k = 0; k < row; k++) {
nodelist.get(k).setProperty("ID", k);
nodelist.get(k).setProperty("Group","Random");
nodelist.get(k).addLabel(myLabel);
}
Relationship rel;
final RelationshipType type2 = DynamicRelationshipType.withName("Rel");
for (int j = 0; j < list.size()-1 ; j += 2) { //list.size()=34420736
rel = nodelist.get(list.get(j)).createRelationshipTo(nodelist.get(list.get(j+1)), type2);
rel.setProperty("Team", "Common");
if (j > 0 && j % 10000 == 0) {// as to commit transaction every now and then and dont throw heap space
tx.success();
tx.close();
tx = graphDb.beginTx();
}
}
tx.success();
}
finally {
tx.close();
}
graphDb.shutdown();
当我运行这段代码时,它抛出以下错误。我使用的是Neo4j 2.3.3和带有Java 8的Netbeans 8.1。我想知道问题是堆空间还是在尝试提交事务时出现的。我还将命令行选项-Xmx1g添加到我的项目中,以增加堆空间。
有任何想法吗?
下面是错误消息:
Exception in thread "main" org.neo4j.graphdb.TransactionFailureException: Transaction was marked as successful, but unable to commit transaction so rolled back.
at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:121)
at com.mycompany.traverse_test.traverse_main.main(traverse_main.java:232)
Caused by: org.neo4j.kernel.api.exceptions.TransactionFailureException: Could not apply the transaction to the store after written to log
at org.neo4j.kernel.impl.api.TransactionRepresentationCommitProcess.applyToStore(TransactionRepresentationCommitProcess.java:105)
at org.neo4j.kernel.impl.api.TransactionRepresentationCommitProcess.commit(TransactionRepresentationCommitProcess.java:58)
at org.neo4j.kernel.impl.api.KernelTransactionImplementation.commit(KernelTransactionImplementation.java:565)
at org.neo4j.kernel.impl.api.KernelTransactionImplementation.close(KernelTransactionImplementation.java:458)
at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:97)
... 1 more
Caused by: java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at org.neo4j.unsafe.impl.internal.dragons.UnsafeUtil.allocateMemory(UnsafeUtil.java:386)
at org.neo4j.unsafe.impl.internal.dragons.MemoryManager$Slab.<init>(MemoryManager.java:111)
at org.neo4j.unsafe.impl.internal.dragons.MemoryManager.allocateAligned(MemoryManager.java:82)
at org.neo4j.io.pagecache.impl.muninn.MuninnPage.initBuffer(MuninnPage.java:417)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:230)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:157)
at org.neo4j.io.pagecache.impl.muninn.MuninnWritePageCursor.next(MuninnWritePageCursor.java:58)
at org.neo4j.kernel.impl.store.PropertyStore.updateRecord(PropertyStore.java:144)
at org.neo4j.kernel.impl.transaction.command.NeoStoreTransactionApplier.visitPropertyCommand(NeoStoreTransactionApplier.java:99)
at org.neo4j.kernel.impl.api.CommandApplierFacade.visitPropertyCommand(CommandApplierFacade.java:120)
at org.neo4j.kernel.impl.transaction.command.Command$PropertyCommand.handle(Command.java:288)
at org.neo4j.kernel.impl.api.CommandApplierFacade.visit(CommandApplierFacade.java:82)
at org.neo4j.kernel.impl.api.CommandApplierFacade.visit(CommandApplierFacade.java:45)
at org.neo4j.kernel.impl.transaction.log.PhysicalTransactionRepresentation.accept(PhysicalTransactionRepresentation.java:69)
at org.neo4j.kernel.impl.api.TransactionRepresentationStoreApplier.apply(TransactionRepresentationStoreApplier.java:111)
at org.neo4j.kernel.impl.api.TransactionRepresentationCommitProcess.applyToStore(TransactionRepresentationCommitProcess.java:100)
... 5 more
if (j > 0 && j % 10000 == 0)
正是做了这件事情,即在创建了10k个关系后提交一次事务。 - dimcode