Databricks-Certified-Data-Engineer-Professional最も有効な質問と解答で勉強

When evaluating the Ganglia Metrics for a given cluster with 3 executor nodes, which indicator would signal proper utilization of the VM's resources?

• A. Bytes Received never exceeds 80 million bytes per second
• B. Network I/O never spikes
• C. The five Minute Load Average remains consistent/flat
• D. Total Disk Space remains constant
• E. CPU Utilization is around 75%

A data engineer manages a production Lakeflow Declarative Pipeline that processes customer transaction data. The pipeline includes several data quality expectations such as transaction_amount > 0 and customer_id IS NOT NULL. These expectations are defined using the EXPECT clause in SQL.
The engineer aims to monitor the pipeline's data quality by analyzing the number of records that passed or failed each expectation during the latest pipeline update. The Lakeflow Declarative Pipelines event logs are stored in a Delta table named event_log_table.
For the most recent pipeline update, determine a programmatically appropriate approach to extract information like the name of each expectation, associated dataset, count of records that passed the expectation, and count of records that failed the expectation.
Which method retrieves the desired data quality metrics from the Lakeflow Declarative Pipelines event log?

• A. Query the event_log_table for events with event_type = 'data_quality' and directly select the passed_records and failed_records fields.
• B. Access the event_log_table, filter for events where event_type = 'flow_progress', and parse details.flow_progress.data_quality.expectations field to extract the required metrics.
• C. Access the event_log_table, filter for events where event_type = 'expectation_result', and extract the expectation metrics from the details field.
• D. Use the Lakeflow Declarative Pipelines UI to navigate to the specific pipeline, select the dataset, and view the Data Quality tab to manually retrieve the expectation metrics.

A Spark job is taking longer than expected. Using the Spark UI, a data engineer notes that the Min, Median, and Max Durations for tasks in a particular stage show the minimum and median time to complete a task as roughly the same, but the max duration for a task to be roughly 100 times as long as the minimum.
Which situation is causing increased duration of the overall job?

• A. Network latency due to some cluster nodes being in different regions from the source data
• B. Skew caused by more data being assigned to a subset of spark-partitions.
• C. Credential validation errors while pulling data from an external system.
• D. Task queueing resulting from improper thread pool assignment.
• E. Spill resulting from attached volume storage being too small.

A data pipeline uses Structured Streaming to ingest data from kafka to Delta Lake. Data is being stored in a bronze table, and includes the Kafka_generated timesamp, key, and value. Three months after the pipeline is deployed the data engineering team has noticed some latency issued during certain times of the day.
A senior data engineer updates the Delta Table's schema and ingestion logic to include the current timestamp (as recoded by Apache Spark) as well the Kafka topic and partition. The team plans to use the additional metadata fields to diagnose the transient processing delays.
Which limitation will the team face while diagnosing this problem?

• A. Updating the table schema will invalidate the Delta transaction log metadata.
• B. New fields cannot be added to a production Delta table.
• C. New fields will not be computed for historic records.
• D. Spark cannot capture the topic partition fields from the kafka source.
• E. Updating the table schema requires a default value provided for each file added.

How are the operational aspects of Lakeflow Declarative Pipelines different from Spark Structured Streaming?

• A. Structured Streaming can process continuous data streams, while Lakeflow Declarative Pipelines cannot.
• B. Lakeflow Declarative Pipelines automatically handle schema evolution, while Structured Streaming always requires manual schema management.
• C. Lakeflow Declarative Pipelines manage the orchestration of multi-stage pipelines automatically, while Structured Streaming requires external orchestration for complex dependencies.
• D. Lakeflow Declarative Pipelines can write to Delta Lake format, while Structured Streaming cannot.