U d@sddlZddlZddlZddlZddlZddlmZdZdZ ddgZ dgZ gZ ddZ dd Zd d Zd d ZddZdddZdS)N)rmtreehadoop3zhive2.3without-hadoopcCs d||fS)zY Check the generated package name, here we need to use the final hadoop version. z %s-bin-%s spark_versionhadoop_version hive_versionrrF/opt/spark-rm/output/spark-3.5.0-bin-hadoop3/python/pyspark/install.pychecked_package_name"sr cCstd|rd|}|ds*td||dkr8d}ntd|rLd|}|tkrjtd |d tftd |r~d |}|tkrtd |d tf|t|||fS)a Check the valid combinations of supported versions in Spark distributions. Parameters ---------- spark_version : str Spark version. It should be X.X.X such as '3.0.0' or spark-3.0.0. hadoop_version : str Hadoop version. It should be X such as '2' or 'hadoop2'. 'without' and 'without-hadoop' are supported as special keywords for Hadoop free distribution. hive_version : str Hive version. It should be X.X such as '2.3' or 'hive2.3'. Parameters ---------- tuple fully-qualified versions of Spark, Hadoop and Hive in a tuple. For example, spark-3.2.0, hadoop3 and hive2.3. z^[0-9]+\.[0-9]+\.[0-9]+$zspark-%szspark-z@Spark version should start with 'spark-' prefix; however, got %swithoutrz^[0-9]+$zhadoop%szOSpark distribution of %s is not supported. Hadoop version should be one of [%s], z^[0-9]+\.[0-9]+$zhive%szMSpark distribution of %s is not supported. Hive version should be one of [%s])rematch startswith RuntimeErrorSUPPORTED_HADOOP_VERSIONSjoinSUPPORTED_HIVE_VERSIONSconvert_old_hadoop_versionrrrr checked_versions)s2      rcCsZddddd}td|}t|d}t|d}|d ksN|d krV|dkrV||}|S) Nz hadoop3.2z hadoop2.7r r)rZhadoop2r rz"^spark-([0-9]+)\.([0-9]+)\.[0-9]+$)rsearchintgroup)rrZ version_dictZspark_version_partsZspark_major_versionZspark_minor_versionrrr r\s rc Cst|||}tj|d|}dtjkr6tjdg}nt}td|d|fd||dkrbdn|f}|D]}tj|dd d |||f} d } zztd || ft t j | |td |t|d} | D]6} | j|krqtj| j|tjj| _| | |qWWDd Stk rVtd|| ftt|ddYnXW5| d k rn| tj |rt |Xqntd|d S)a Installs Spark that corresponds to the given Hadoop version in the current library directory. Parameters ---------- dest : str The location to download and install the Spark. spark_version : str Spark version. It should be spark-X.X.X form. hadoop_version : str Hadoop version. It should be hadoopX.X such as 'hadoop2.7' or 'without-hadoop'. hive_version : str Hive version. It should be hiveX.X such as 'hive2.3'. z%s.tgzZPYSPARK_RELEASE_MIRRORz%Trying to download Spark %s from [%s]r z%s for Hadoop %sr z Free buildT)exist_okz%s/spark/%s/%s.tgzNzDownloading %s from: - %szInstalling to %szr:gzzFailed to download %s from %s:) ignore_errorszUnable to download %s.)r ospathrenvironget_preferred_mirrorsprintmakedirscloseexistsremovedownload_to_fileurllibrequesturlopentarfileopen getmembersnamerelpathsepextract Exception traceback print_excrIOError) destrrr package_nameZpackage_local_pathZsitesZpretty_pkg_namesiteurltarmemberrrr install_sparklsD         r=c stgtdD]>}z$tjd}|dWq tk rHYq Xq dddg}tt fdd|DS) Nrz4https://www.apache.org/dyn/closer.lua?preferred=truezutf-8zhttps://dlcdn.apache.org/zhttps://archive.apache.org/distz*https://dist.apache.org/repos/dist/releasecsg|]}|kr|qSrr).0xZ mirror_urlsrr sz)get_preferred_mirrors..) ranger)r*r+appendreaddecoder3listset)_responseZ default_sitesrr@r r"s r"c Cst|d}d}t|ddR}||}|t|7}|sDqt||td||t t ||ddfq(W5QRXdS)NzContent-Lengthrwb)modez#Downloaded %d of %d bytes (%0.2f%%)dr) rinfogetstripr-rDlenwriter#roundfloat)rIr chunk_sizeZ total_sizeZ bytes_so_farr7chunkrrr r(s   r()rJ)rrr,r4Zurllib.requestr)shutilrDEFAULT_HADOOP DEFAULT_HIVErrZUNSUPPORTED_COMBINATIONSr rrr=r"r(rrrr s  3=