我有一个非常大(11GB)的JSON文件,太大了无法读入内存。我想将它拆分成较小的文件以便分析数据。我目前正在使用Python和Pandas进行分析,我想知道是否有一种方法可以访问文件的块,使其可以在不崩溃程序的情况下被读入内存。理想情况下,我想将一年的数据拆分成更小的可管理文件,每个文件跨度约为一周,但是数据大小不是恒定的,尽管如果它们是固定间隔也没关系。
以下是数据格式:
以下是数据格式:
{
"actor" :
{
"classification" : [ "suggested" ],
"displayName" : "myself",
"followersCount" : 0,
"followingCount" : 0,
"followingStocksCount" : 0,
"id" : "person:stocktwits:183087",
"image" : "http://avatars.stocktwits.com/production/183087/thumb-1350332393.png",
"link" : "http://stocktwits.com/myselfbtc",
"links" :
[
{
"href" : null,
"rel" : "me"
}
],
"objectType" : "person",
"preferredUsername" : "myselfbtc",
"statusesCount" : 2,
"summary" : null,
"tradingStrategy" :
{
"approach" : "Technical",
"assetsFrequentlyTraded" : [ "Forex" ],
"experience" : "Novice",
"holdingPeriod" : "Day Trader"
}
},
"body" : "$BCOIN and macd is going down ..... http://stks.co/iDEB",
"entities" :
{
"chart" :
{
"fullImage" :
{
"link" : "http://charts.stocktwits.com/production/original_10047145.png"
},
"image" :
{
"link" : "http://charts.stocktwits.com/production/small_10047145.png"
},
"link" : "http://stks.co/iDEB",
"objectType" : "image"
},
"sentiment" :
{
"basic" : "Bearish"
},
"stocks" :
[
{
"displayName" : "Bitcoin",
"exchange" : "PRIVATE",
"industry" : null,
"sector" : null,
"stocktwits_id" : 9659,
"symbol" : "BCOIN"
}
],
"video" : null
},
"gnip" :
{
"language" :
{
"value" : "en"
}
},
"id" : "tag:gnip.stocktwits.com:2012:note/10047145",
"inReplyTo" :
{
"id" : "tag:gnip.stocktwits.com:2012:note/10046953",
"objectType" : "comment"
},
"link" : "http://stocktwits.com/myselfbtc/message/10047145",
"object" :
{
"id" : "note:stocktwits:10047145",
"link" : "http://stocktwits.com/myselfbtc/message/10047145",
"objectType" : "note",
"postedTime" : "2012-10-17T19:13:50Z",
"summary" : "$BCOIN and macd is going down ..... http://stks.co/iDEB",
"updatedTime" : "2012-10-17T19:13:50Z"
},
"provider" :
{
"displayName" : "StockTwits",
"link" : "http://stocktwits.com"
},
"verb" : "post"
}
c4.4xlarge
,配备32GB RAM,每小时不到1美元。而且那还不是他们最大的机器。谷歌也在他们的云中提供一些大内存VMs出租,我记得它们更贵。学会如何在不使用时完全删除VM,这样你就不会因为空闲时间而被计费,并且当这些问题出现时,你将始终可以访问一个大型机器。 - Paul