[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    Re: Dependency on absolute path /bin/ls in Shell.java
From:       Namikaze Minato <lloydsensei () gmail ! com>
Date:       2015-12-12 11:29:22
Message-ID: CACmJb3zXKG4HD4dKSJf+N8Q4_mNLTD9qgWn9RVWfkz+uLirA5g () mail ! gmail ! com
[Download RAW message or body]

Then it is enough :)

Regards,
LLoyd

On 11 December 2015 at 23:27, Samuel <samuelrivas@gmail.com> wrote:
> > My question was, which spark command are you using, and since you
> > already did the analysis, which function of Shell.java is this spark
> > code using?
> 
> Sorry, I misunderstood you. It was something using RawLocalFileSystem
> to load parquet files. The problem seemed to go away after I upgraded
> to spark 1.5.2 (that was in spark 1.4.0).
> 
> I don't have the code with me now, but if the information is useful in
> any way I can provide an example later.
> 
> Otherwise, there seems to be a ticket already to reimplement that code
> path, so that is good enough for me.
> 
> Best
> 
> > 
> > Regards,
> > LLoyd
> > 
> > On 11 December 2015 at 15:43, Samuel <samuelrivas@gmail.com> wrote:
> > > I am not using hadoop-util directly, it is Spark code what uses it
> > > (i,e. not directly under my control).
> > > 
> > > Regarding ls, for my particular use case it is fine if you use "ls"
> > > instead of "/bin/ls".
> > > 
> > > However, I do agree that using ls to fetch file permissions is
> > > incorrect, so a better solution (in terms of code quality) would be
> > > not to use ls at all.
> > > 
> > > 
> > > On 11 December 2015 at 14:56, Namikaze Minato <lloydsensei@gmail.com> wrote:
> > > > So what you ultimately need is a piece of java code listing the rwx
> > > > permissions for user, group and others that is not using ls
> > > > internally, is that correct?
> > > > If "RawLocalFileSystem" is not HDFS, do you really need to use
> > > > hadoop-util for that?
> > > > Can you tell us more about your use case?
> > > > 
> > > > Regards,
> > > > LLoyd
> > > > 
> > > > 
> > > > 
> > > > 
> > > > On 11 December 2015 at 13:05, Samuel <samuelrivas@gmail.com> wrote:
> > > > > > Using ls to figure out permissions is a bad design anyway, so I would
> > > > > > not be surprised if this hardcode was reported as a bug.
> > > > > 
> > > > > Of course, I have no idea why it was implemented like this. I assume
> > > > > it was written at some point in time where Java didn't provide the
> > > > > needed APIS (?)
> > > > > 
> > > > > Implementing the permission check without relying in ls at all is also
> > > > > a solution for the problem I have :)
> > > > > 
> > > > > > LLoyd
> > > > > > 
> > > > > > On 11 December 2015 at 09:19, Samuel <samuelrivas@gmail.com> wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > > > I am experiencing some crashes when using spark over local files \
> > > > > > > (mainly for testing). Some operations fail with
> > > > > > > 
> > > > > > > java.lang.RuntimeException: Error while running command to get file
> > > > > > > permissions : java.io.IOException: Cannot run program "/bin/ls": \
> > > > > > > error=2, No such file or directory
> > > > > > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
> > > > > > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
> > > > > > > at org.apache.hadoop.util.Shell.run(Shell.java:188)
> > > > > > > at
> > > > > > > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
> > > > > > >  at org.apache.hadoop.util.Shell.execCommand(Shell.java:467)
> > > > > > > at org.apache.hadoop.util.Shell.execCommand(Shell.java:450)
> > > > > > > at
> > > > > > > org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:593)
> > > > > > >  at
> > > > > > > org.apache.hadoop.fs.RawLocalFileSystem.access$100(RawLocalFileSystem.java:51)
> > > > > > >  at
> > > > > > > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:514)
> > > > > > >  at
> > > > > > > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:489)
> > > > > > >  at
> > > > > > > org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$buildScan$1$$anon$1$$anonfun$12.apply(newParquet.scala:292)
> > > > > > >  
> > > > > > > etcetera...
> > > > > > > 
> > > > > > > Which seems to be related to Shell.java in org.apache.hadoop-util, that \
> > > > > > > uses ls -ld to figure out file permissions (that is in
> > > > > > > RawLocalFileSystem.loadPermissionsInfo). The problem is that instead of \
> > > > > > > just calling ls, Shell .java calls /bin/ls, which is usually available, \
> > > > > > > but in certain circumstances might not. Regardless of the reasons not \
> > > > > > > to have ls in /bin, hardcoding the directory bans users from using the \
> > > > > > > standard mechanisms to decide which binaries to run in their systems \
> > > > > > > (in this case, $PATH), so I wonder if there is a particular reason why \
> > > > > > > that path has been hardcoded to an absolute path instead to something \
> > > > > > > resolvable using$PATH. 
> > > > > > > Or in other words, is this a bug or a feature?
> > > > > > > 
> > > > > > > Best
> > > > > > > 
> > > > > > > --
> > > > > > > Samuel
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > --
> > > > > Samuel
> > > 
> > > 
> > > 
> > > --
> > > Samuel
> 
> 
> 
> 
> --
> Samuel

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org


[prev in list] [next in list] [prev in thread] [next in thread]