pyspark-shell - Details for Query 13

Submitted Time: 2026/05/01 06:32:07
Duration: 1 s
Succeeded Jobs: 4

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { subgraph cluster0 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 4 ms"; 1 [labelType="html" label=" Project "]; 2 [labelType="html" label=" GlobalLimit "]; } 3 [labelType="html" label="Exchange shuffle records written: 40 shuffle write time total (min, med, max (stageId: taskId)) 19 ms (0 ms, 0 ms, 9 ms (stage 4.0: task 5)) records read: 5 local bytes read: 6.8 KiB fetch wait time: 0 ms remote bytes read: 1043.0 B local blocks read: 7 remote blocks read: 1 data size total (min, med, max (stageId: taskId)) 11.6 KiB (1472.0 B, 1480.0 B, 1496.0 B (stage 4.0: task 8)) shuffle bytes written total (min, med, max (stageId: taskId)) 7.8 KiB (922.0 B, 1004.0 B, 1043.0 B (stage 4.0: task 4))"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n1.7 s (15 ms, 32 ms, 1.1 s (stage 4.0: task 4))"; 5 [labelType="html" label=" LocalLimit "]; 6 [labelType="html" label="ColumnarToRow number of output rows: 32,768 number of input batches: 8"]; } 7 [labelType="html" label="Scan parquet itv023599_lending_club.customers number of files read: 200 scan time total (min, med, max (stageId: taskId)) 1.7 s (15 ms, 31 ms, 1.1 s (stage 4.0: task 4)) metadata time: 0 ms size of files read: 184.8 MiB number of output rows: 32,768"]; 2->1; 3->2; 5->3; 6->5; 7->6; }

Project [member_id#279, emp_title#280, cast(emp_length#281 as string) AS emp_length#345, home_ownership#282, cast(annual_income#283 as string) AS annual_income#347, address_state#284, address_zipcode#285, address_country#286, grade#287, sub_grade#288, verification_status#289, cast(total_high_credit_limit#290 as string) AS total_high_credit_limit#354, application_type#291, cast(join_annual_income#292 as string) AS join_annual_income#356, verification_status_joint#293, cast(ingest_date#294 as string) AS ingest_date#358]

GlobalLimit 5

WholeStageCodegen (2)

Exchange SinglePartition, ENSURE_REQUIREMENTS, [id=#93]

LocalLimit 5

ColumnarToRow

WholeStageCodegen (1)

FileScan parquet itv023599_lending_club.customers[member_id#279,emp_title#280,emp_length#281,home_ownership#282,annual_income#283,address_state#284,address_zipcode#285,address_country#286,grade#287,sub_grade#288,verification_status#289,total_high_credit_limit#290,application_type#291,join_annual_income#292,verification_status_joint#293,ingest_date#294] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[hdfs://m01.itversity.com:9000/user/itv023599/lendingclubproject/cleaned/custome..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<member_id:string,emp_title:string,emp_length:int,home_ownership:string,annual_income:float...

Details

== Physical Plan ==
* Project (6)
+- * GlobalLimit (5)
   +- Exchange (4)
      +- * LocalLimit (3)
         +- * ColumnarToRow (2)
            +- Scan parquet itv023599_lending_club.customers (1)


(1) Scan parquet itv023599_lending_club.customers
Output [16]: [member_id#279, emp_title#280, emp_length#281, home_ownership#282, annual_income#283, address_state#284, address_zipcode#285, address_country#286, grade#287, sub_grade#288, verification_status#289, total_high_credit_limit#290, application_type#291, join_annual_income#292, verification_status_joint#293, ingest_date#294]
Batched: true
Location: InMemoryFileIndex [hdfs://m01.itversity.com:9000/user/itv023599/lendingclubproject/cleaned/customers_parquet]
ReadSchema: struct<member_id:string,emp_title:string,emp_length:int,home_ownership:string,annual_income:float,address_state:string,address_zipcode:string,address_country:string,grade:string,sub_grade:string,verification_status:string,total_high_credit_limit:float,application_type:string,join_annual_income:float,verification_status_joint:string,ingest_date:timestamp>

(2) ColumnarToRow [codegen id : 1]
Input [16]: [member_id#279, emp_title#280, emp_length#281, home_ownership#282, annual_income#283, address_state#284, address_zipcode#285, address_country#286, grade#287, sub_grade#288, verification_status#289, total_high_credit_limit#290, application_type#291, join_annual_income#292, verification_status_joint#293, ingest_date#294]

(3) LocalLimit [codegen id : 1]
Input [16]: [member_id#279, emp_title#280, emp_length#281, home_ownership#282, annual_income#283, address_state#284, address_zipcode#285, address_country#286, grade#287, sub_grade#288, verification_status#289, total_high_credit_limit#290, application_type#291, join_annual_income#292, verification_status_joint#293, ingest_date#294]
Arguments: 5

(4) Exchange
Input [16]: [member_id#279, emp_title#280, emp_length#281, home_ownership#282, annual_income#283, address_state#284, address_zipcode#285, address_country#286, grade#287, sub_grade#288, verification_status#289, total_high_credit_limit#290, application_type#291, join_annual_income#292, verification_status_joint#293, ingest_date#294]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [id=#93]

(5) GlobalLimit [codegen id : 2]
Input [16]: [member_id#279, emp_title#280, emp_length#281, home_ownership#282, annual_income#283, address_state#284, address_zipcode#285, address_country#286, grade#287, sub_grade#288, verification_status#289, total_high_credit_limit#290, application_type#291, join_annual_income#292, verification_status_joint#293, ingest_date#294]
Arguments: 5

(6) Project [codegen id : 2]
Output [16]: [member_id#279, emp_title#280, cast(emp_length#281 as string) AS emp_length#345, home_ownership#282, cast(annual_income#283 as string) AS annual_income#347, address_state#284, address_zipcode#285, address_country#286, grade#287, sub_grade#288, verification_status#289, cast(total_high_credit_limit#290 as string) AS total_high_credit_limit#354, application_type#291, cast(join_annual_income#292 as string) AS join_annual_income#356, verification_status_joint#293, cast(ingest_date#294 as string) AS ingest_date#358]
Input [16]: [member_id#279, emp_title#280, emp_length#281, home_ownership#282, annual_income#283, address_state#284, address_zipcode#285, address_country#286, grade#287, sub_grade#288, verification_status#289, total_high_credit_limit#290, application_type#291, join_annual_income#292, verification_status_joint#293, ingest_date#294]