pyspark-shell - Details for Query 2

Submitted Time: 2026/05/09 01:00:21
Duration: 3 s
Succeeded Jobs: 2 3 4

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" CollectLimit "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n6.2 s (248 ms, 405 ms, 1.8 s (stage 5.0: task 261))"; 2 [labelType="html" label=" Project "]; 3 [labelType="html" label="Filter number of output rows: 5"]; 4 [labelType="html" label="ColumnarToRow number of output rows: 2,260,633 number of input batches: 600"]; } 5 [labelType="html" label="Scan parquet itv024694_lending_club.customers number of files read: 200 scan time total (min, med, max (stageId: taskId)) 5.8 s (222 ms, 356 ms, 1.8 s (stage 5.0: task 261)) metadata time: 0 ms size of files read: 184.9 MiB number of output rows: 2,260,633"]; 2->0; 3->2; 4->3; 5->4; }

CollectLimit 21

Project [member_id#56, emp_title#57, cast(emp_length#58 as string) AS emp_length#106, home_ownership#59, cast(annual_income#60 as string) AS annual_income#108, address_state#61, address_zipcode#62, address_country#63, grade#64, sub_grade#65, verification_status#66, cast(total_high_credit_limit#67 as string) AS total_high_credit_limit#115, application_type#68, cast(join_annual_income#69 as string) AS join_annual_income#117, verification_status_joint#70, cast(ingest_date#71 as string) AS ingest_date#119]

Filter (isnotnull(member_id#56) AND StartsWith(member_id#56, e4c167053d5418230))

ColumnarToRow

WholeStageCodegen (1)

FileScan parquet itv024694_lending_club.customers[member_id#56,emp_title#57,emp_length#58,home_ownership#59,annual_income#60,address_state#61,address_zipcode#62,address_country#63,grade#64,sub_grade#65,verification_status#66,total_high_credit_limit#67,application_type#68,join_annual_income#69,verification_status_joint#70,ingest_date#71] Batched: true, DataFilters: [isnotnull(member_id#56), StartsWith(member_id#56, e4c167053d5418230)], Format: Parquet, Location: InMemoryFileIndex[hdfs://m01.itversity.com:9000/public/trendytech/lendingclubproject/cleaned/cust..., PartitionFilters: [], PushedFilters: [IsNotNull(member_id), StringStartsWith(member_id,e4c167053d5418230)], ReadSchema: struct<member_id:string,emp_title:string,emp_length:int,home_ownership:string,annual_income:float...

Details

== Physical Plan ==
CollectLimit (5)
+- * Project (4)
   +- * Filter (3)
      +- * ColumnarToRow (2)
         +- Scan parquet itv024694_lending_club.customers (1)


(1) Scan parquet itv024694_lending_club.customers
Output [16]: [member_id#56, emp_title#57, emp_length#58, home_ownership#59, annual_income#60, address_state#61, address_zipcode#62, address_country#63, grade#64, sub_grade#65, verification_status#66, total_high_credit_limit#67, application_type#68, join_annual_income#69, verification_status_joint#70, ingest_date#71]
Batched: true
Location: InMemoryFileIndex [hdfs://m01.itversity.com:9000/public/trendytech/lendingclubproject/cleaned/customers_parquet]
PushedFilters: [IsNotNull(member_id), StringStartsWith(member_id,e4c167053d5418230)]
ReadSchema: struct<member_id:string,emp_title:string,emp_length:int,home_ownership:string,annual_income:float,address_state:string,address_zipcode:string,address_country:string,grade:string,sub_grade:string,verification_status:string,total_high_credit_limit:float,application_type:string,join_annual_income:float,verification_status_joint:string,ingest_date:timestamp>

(2) ColumnarToRow [codegen id : 1]
Input [16]: [member_id#56, emp_title#57, emp_length#58, home_ownership#59, annual_income#60, address_state#61, address_zipcode#62, address_country#63, grade#64, sub_grade#65, verification_status#66, total_high_credit_limit#67, application_type#68, join_annual_income#69, verification_status_joint#70, ingest_date#71]

(3) Filter [codegen id : 1]
Input [16]: [member_id#56, emp_title#57, emp_length#58, home_ownership#59, annual_income#60, address_state#61, address_zipcode#62, address_country#63, grade#64, sub_grade#65, verification_status#66, total_high_credit_limit#67, application_type#68, join_annual_income#69, verification_status_joint#70, ingest_date#71]
Condition : (isnotnull(member_id#56) AND StartsWith(member_id#56, e4c167053d5418230))

(4) Project [codegen id : 1]
Output [16]: [member_id#56, emp_title#57, cast(emp_length#58 as string) AS emp_length#106, home_ownership#59, cast(annual_income#60 as string) AS annual_income#108, address_state#61, address_zipcode#62, address_country#63, grade#64, sub_grade#65, verification_status#66, cast(total_high_credit_limit#67 as string) AS total_high_credit_limit#115, application_type#68, cast(join_annual_income#69 as string) AS join_annual_income#117, verification_status_joint#70, cast(ingest_date#71 as string) AS ingest_date#119]
Input [16]: [member_id#56, emp_title#57, emp_length#58, home_ownership#59, annual_income#60, address_state#61, address_zipcode#62, address_country#63, grade#64, sub_grade#65, verification_status#66, total_high_credit_limit#67, application_type#68, join_annual_income#69, verification_status_joint#70, ingest_date#71]

(5) CollectLimit
Input [16]: [member_id#56, emp_title#57, emp_length#106, home_ownership#59, annual_income#108, address_state#61, address_zipcode#62, address_country#63, grade#64, sub_grade#65, verification_status#66, total_high_credit_limit#115, application_type#68, join_annual_income#117, verification_status_joint#70, ingest_date#119]
Arguments: 21