[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    Run job not from namenode
From:       Andrey Pankov <apankov () iponweb ! net>
Date:       2008-03-31 11:57:39
Message-ID: 47F0D1B3.1070005 () iponweb ! net
[Download RAW message or body]

Hi all,

Currently I'm able to run map-reduce jobs from box where NameNode and 
JobTracker are running. But I'd like to run my jobs from separate box, 
from which I have access to HDFS. I have updated params fs.default.name 
and  mapred.job.tracker in local hadoop dir to point to the clusters 
master. Now Hadoop returns me following error:

username@ip-10-251-96-129:/usr/local/hadoop-0.16.0$ bin/hadoop jar 
hadoop-0.16.0-examples.jar wordcount /user/username/gutenberg 
/user/username/gutenberg-output
08/03/31 10:21:46 INFO mapred.FileInputFormat: Total input paths to 
process : 3
org.apache.hadoop.ipc.RemoteException: java.io.IOException: 
/mnt/hadoop/mapred/system/job_200803210640_0852/job.xml: No such file or 
directory
         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:159)
         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:133)
         at 
org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1083)
  ...

Here account 'username' has passwordless access to master box. Cluster 
runs over EC2.

As a variant I can run tasks via ssh, i.e.

ssh master /usr/local/hadoop-0.16.0bin/hadoop jar 
/home/username/jobs/hadoop-0.16.0-examples.jar wordcount 
/user/username/gutenberg /user/username/gutenberg-output

But you need to put your jar file to the NameNode box before you run it.

Thanks in advance.

-- 
Andrey Pankov

[prev in list] [next in list] [prev in thread] [next in thread]