[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-dev
Subject:    Re: CredentialProvider API
From:       Wei-Chiu Chuang <weichiu () cloudera ! com ! INVALID>
Date:       2019-04-25 14:46:17
Message-ID: CADiq6=yLm-WHE7RDx9t=QYrpNhfsKZEQ9XW88nJ=vvz-NS+Oig () mail ! gmail ! com
[Download RAW message or body]


A similar issue is https://issues.apache.org/jira/browse/HADOOP-15412
You can't put the keystore for KMS in HDFS, and I filed
https://issues.apache.org/jira/browse/HADOOP-15413 to document it.

Karthik, would you consider use HADOOP-15413 to document scenarios where it
is not supported?

Additionally, I worked on HADOOP-13548
<https://issues.apache.org/jira/browse/HADOOP-13548> awhile ago to remove
the circular dependency for the LdapGroupsMapping case.
You can also consider taking over to resolve this case.

On Thu, Apr 25, 2019 at 3:20 AM Steve Loughran <stevel@cloudera.com.invalid>
wrote:

> didn't think it worked at all, that is: you can't store the credentials to
> access an FS in the FS. There's some configuration option to disable
> specific sources of JCEKS files, and the one which reads in through the
> hadoop FS API shouldn't be included when reading any credentials needed to
> work with HDFS.
> 
> JCEKS files in HDFS work for storing object storage credentials.
> 
> On Thu, Apr 25, 2019 at 3:13 AM larry mccay <larry.mccay@gmail.com> wrote:
> 
> > This is likely an issue only for issues where we need the password from
> > HDFS in order to access HDFS.
> > This should definitely be avoided by not having a static credential
> > provider path configured for startup that includes such a dependency.
> > 
> > For instance, the JIRA you cite is an example where we need to do group
> > lookup in order to determine whether you are allowed to access the HDFS
> > resource that provides the password required to do group lookup.
> > 
> > Storing passwords in credential stores within HDFS should be perfectly
> safe
> > for things like SSL that don't have a dependency on HDFS itself.
> > 
> > Those details are in the documentation page that you referenced but if
> they
> > need to be made more clear that completely makes sense.
> > 
> > On Wed, Apr 24, 2019 at 9:56 PM Karthik P <karthikshva123@gmail.com>
> > wrote:
> > 
> > > Team,
> > > 
> > > Datanode is failed to restart after configuring credentials provider,
> > > storing credential into HDFS (jceks://hdfs@hostname
> > > > 9001/credential/keys.jceks).
> > > 
> > > Getting a StackOverFlow error in datanode jsvc.out file similar to
> > > HADOOP-11934 <https://issues.apache.org/jira/browse/HADOOP-11934>.
> > > 
> > > As per the documentation link
> > > <
> > > 
> > 
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Supported_Features
> 
> > > > ,
> > > we support storing credential in HDFS.
> > > 
> > > *URI jceks://file|hdfs/path-to-keystore, is used to retrieve
> credentials
> > > from a Java keystore. The underlying use of the Hadoop filesystem
> > > abstraction allows credentials to be stored on the local filesystem or
> > > within HDFS.*
> > > 
> > > Assume a scenario, where all of our data nodes were down and we
> > configured
> > > hadoop.security.credential.provider.path to HDFS location. So when we
> try
> > > to get FileSystem.get() during datanode restart we end up doing
> recursive
> > > call if HDFS is inaccessible.
> > > 
> > > 
> > > /**
> > > * Check and set 'configuration' if necessary.
> > > *
> > > * @param theObject object for which to set configuration
> > > * @param conf Configuration
> > > */
> > > public static void setConf(Object theObject, Configuration conf) {
> > > if (conf != null) {
> > > if (theObject instanceof Configurable) {
> > > ((Configurable) theObject).setConf(conf);
> > > }
> > > setJobConf(theObject, conf);
> > > }
> > > }
> > > 
> > > 
> > > No issues if we store credential in LFS (localjceks://file). The
> problem
> > > only with jceks://hdfs/.
> > > 
> > > Can I change Hadoop doc that we would not support storing credential in
> > > HDFS? Or Shall I handle this scenario only for statup issue?
> > > 
> > > 
> > > Thanks,
> > > Karthik
> > > 
> > 
> 



[prev in list] [next in list] [prev in thread] [next in thread]