我有一个谷歌应用引擎应用程序，数据存储在谷歌云数据存储中。我想使用Dataflow将部分数据放入BigQuery，但我打算从仅获取一些来自Datastore的信息并将其写入Google Cloud Storage开始。我的代码如下：
public class DatastorePipeline {
private static final Logger LOG = LoggerFactory.getLogger(DatastorePipeline.class);
static class GetEmailFn extends DoFn<Entity, String> {
@Override
public void processElement(ProcessContext c) throws Exception {
Map<String, Value> properties = DatastoreHelper.getPropertyMap(c.element());
Value value = properties.get("email_address");
if(value != null) {
c.output(DatastoreHelper.getString(value));
}
}
}
public static void main(String[] args) {
Pipeline p = Pipeline.create(PipelineOptionsFactory.fromArgs(args).withValidation().create());
Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("User");
Query query = q.build();
DatastoreIO.Source source = DatastoreIO.source()
.withDataset("my-project-id")
.withQuery(query);
p.apply("ReadUsersFromDatastore", Read.from(source))
.apply(ParDo.named("GetEmailAddress").of(new GetEmailFn()))
.apply(TextIO.Write.to("gs://dataflow-output-bucket/emails.txt"));
p.run();
}
}
然而，当我尝试运行这个代码时，我在执行Datastore查询时遇到了403错误：
Request failed with code 403, will NOT retry: https://www.googleapis.com/datastore/v1beta2/datasets/my-project-id/runQuery
我正在使用Google Cloud Dataflow插件从Eclipse运行此程序。在没有Datastore读取的数据流作业中运行正常。我进行了
gcloud auth login
在运行作业之前，需要像教程中描述的那样进行配置。我做错了什么？
编辑：下面是完整的堆栈跟踪。
Oct 11, 2015, 12:03:13 PM (b6119cca307b4d9a): com.google.api.services.datastore.client.DatastoreException: Unauthorized. at com.google.api.services.datastore.client.RemoteRpc.makeException(RemoteRpc.java:115) at com.google.api.services.datastore.client.RemoteRpc.call(RemoteRpc.java:81) at com.google.api.services.datastore.client.BaseDatastoreFactory$RemoteRpc.call(BaseDatastoreFactory.java:41) at com.google.api.services.datastore.client.Datastore.runQuery(Datastore.java:109) at com.google.api.services.datastore.client.QuerySplitterImpl.getScatterKeys(QuerySplitterImpl.java:189) at com.google.api.services.datastore.client.QuerySplitterImpl.getSplits(QuerySplitterImpl.java:75) at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.getSplitQueries(DatastoreIO.java:427) at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.splitIntoBundles(DatastoreIO.java:306) at com.google.cloud.dataflow.sdk.runners.dataflow.BasicSerializableSourceFormat.performSplit(BasicSerializableSourceFormat.java:318) at com.google.cloud.dataflow.sdk.runners.dataflow.BasicSerializableSourceFormat.performSourceOperation(BasicSerializableSourceFormat.java:167) at com.google.cloud.dataflow.sdk.runners.worker.SourceOperationExecutor.execute(SourceOperationExecutor.java:80) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.executeWork(DataflowWorker.java:257) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.doWork(DataflowWorker.java:193) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:146) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.doWork(DataflowWorkerHarness.java:164) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.call(DataflowWorkerHarness.java:145) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.call(DataflowWorkerHarness.java:132) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
答案：问题在于我的项目基于公司的域名限制访问，这是导致服务帐户无法连接的原因。感谢Dan帮助我解决了这个问题！
DataflowPipelineRunner（或者是Blocking变体）。是否尝试使用DirectPipelineRunner运行，看看是否会出现相同的错误？ - Dan Halperin