Google Professional Data Engineer Pipelines and Processing

System Lag vs. Data Freshness: System lag is low (5 seconds), indicating that individual messages are processed quickly. However, data freshness is high (40 seconds), suggesting a backlog in the pipeline.
Not Advertising's Fault: The issue is upstream of their consumption, as they're already receiving delayed messages.
Not Web Server's Fault: The lag between eventTimestamp and publishTime is minimal (1 second), meaning the server is publishing messages promptly.

- It suggest a backlog problem.
- It indicates that while individual messages might be processed quickly once they're handled, the job overall cannot keep up with the rate of incoming messages, causing a delay in processing the backlog.

Low System Lag means no backlog problem. The bottleneck is in the actual processing logic.

Answer is C lol

I don't know why you guys got the processing time is less than 30 sec. But I would consider the processing time with 40(freshness) - 5(system lag) = 35 sec. Even minus the publish time of Pub/sub which is less than 1 sec. The processing time still larger than 30 sec. I believe inspecting a few messages show no more than 1 sec lag is about pub/sub processing time. Not inspecting a few messages for dataflow. So I would choose B.

Option C

Option C - low system lag (which identifies fast processing) but high data freshness (which identifies that the messages sit in the backlog a lot)

agree correct is C

- Dataflow service agent is the one responsible for setting up and managing the network resources that Dataflow requires.
- By granting the compute.networkUser role to this service agent, we are enabling it to provision the necessary network resources within the Shared VPC for your Dataflow job.

All projects that have used the resource Dataflow Job have a Dataflow Service Account, also known as the Dataflow service agent.

Make sure the Shared VPC subnetwork is shared with the Dataflow service account and has the Compute Network User role assigned on the specified subnet.

If VMs need to attach to a subnet :
Grant compute.networkUser to the runtime service account, not the service agent

A Dataflow service agent is the one responsible for setting up and managing the network resources that Dataflow requires.

Key Technical Takeaway
When working with Dataflow and Shared VPC, always remember: Service Agent = Network Permissions (Host Project) and Worker Service Account = Data Permissions (Service Project).

Make sure the Shared VPC subnetwork is shared with the Dataflow service account and has the Compute Network User role assigned on the specified subnet. The Compute Network User role must be assigned to the Dataflow service account in the host project.
https://cloud.google.com/dataflow/docs/guides/specifying-networks?utm_source=chatgpt.com#shared

It’s a

Answer is B:

The Dataflow service agent manages Dataflow internals, but does not launch pipeline worker VMs. So Answer A is incorrect

https://cloud.google.com/dataflow/docs/concepts/security-and-permissions#df-service-account

From https://cloud.google.com/dataflow/docs/guides/specifying-networks, it says "Make sure the Shared VPC subnetwork is shared with the Dataflow service account and has the Compute Network User role assigned on the specified subnet. The Compute Network User role must be assigned to the Dataflow service account in the host project."

If you see in the comments, A was answer by people around 8 months ago but recent ones have answered B with the documentation. The GCP documentation evolves with time

service account that executes the Dataflow pipeline
It's straight forward

Assign the compute.networkUser role to the service account that executes the Dataflow pipeline

The correct answer is B. Assign the compute.networkUser role to the service account that executes the Dataflow pipeline.

Here's why:

Shared VPC and Network Access: When using a Shared VPC, you need to grant specific permissions to service accounts in the service project (where your Dataflow pipeline runs) to access resources in the host project's network.
compute.networkUser Role: This role grants the necessary permissions for a service account to use the network resources in the Shared VPC. This includes accessing subnets, creating instances, and communicating with other services within the network.
Service Account for Pipeline Execution: The service account that executes your Dataflow pipeline is the one that needs these network permissions. This is because the Dataflow service uses this account to create and manage worker instances within the Shared VPC network.

Dataflow service agent is a role that is assigned to a service account. So is compute.networkUser.
https://cloud.google.com/dataflow/docs/concepts/access-control#example

Option B https://cloud.google.com/knowledge/kb/dataflow-job-in-shared-vpc-xpn-permissions-000004261

I believe the answer is B. All authentication documentation points to Service Accounts. https://cloud.google.com/dataflow/docs/concepts/authentication#on-gcp

Dataflow service agent typically manages general interactions with the Dataflow service but does not execute the actual jobs.

D. 1. Use a dual-region Cloud Storage bucket with turbo replication enabled.
2. Monitor Dataflow metrics with Cloud Monitoring to determine when an outage occurs.
3. Seek the subscription back in time by 60 minutes to recover the acknowledged messages.
4. Start the Dataflow job in a secondary region.

RPO of 15 minutes is guaranteed when turbo replication is used
https://cloud.google.com/storage/docs/availability-durability

Option D is correct.

Not A, because dual-region bucket WITHOUT turbo replication takes atleast 1 hour to sync data between regions. SLA for 100% data sync is 12 hours as per google.

it's D, keyword "Turbo replication" for RPO recovery

Could be A or D. The choice between a 15-minute seek and a 60-minute seek depends on your specific requirements and priorities. If a very low RPO is critical, a 60-minute seek might be necessary to ensure data completeness.If minimizing cost and processing time is more important, a 15-minute seek might be sufficient, especially if you're confident in the reliability of Turbo Replication.

An RPO of 15 minutes seemingly suggests using Turbo Replication. But here's the thing - why would you want to seek the subscription back in time by 60 minutes and run the Dataflow job? Thus, if turbo replication is enabled, steps 3 & 4 are completely redundant and unnecessary. Which is why option A is correct. This was a tricky one!

I don't like answer D. If we have turbo replication can ensure that change within 15min can be replicated, why do we still need to seek the subscription back in time by 60min?

D
https://cloud.google.com/storage/docs/availability-durability#cross-region-redundancy

https://cloud.google.com/storage/docs/availability-durability#turbo-replication says : "When enabled, turbo replication is designed to replicate 100% of newly written objects to both regions that constitute the dual-region within the recovery point objective of 15 minutes, regardless of object size."
so seems D to me

- Low RPO: Dual-region buckets offer synchronous replication, ensuring data is immediately available in both regions, aligning with the 15-minute RPO.
- Turbo Replication: enabling turbo replication can further reduce replication latency to near-real-time for even stricter RPO requirements.
- Resilient Data Storage: Dual-region buckets ensure data availability even during regional outages, protecting processed data.
- Fast Recovery: Reprocessing from the last 15 minutes of acknowledged messages minimizes data loss and downtime.

A.
1. Use a dual-region Cloud Storage bucket.
2. Monitor Dataflow metrics with Cloud Monitoring to determine when an outage occurs.
3. Seek the subscription back in time by 15 minutes to recover the acknowledged messages.
4. Start the Dataflow job in a secondary region.

DE

Another way to replay messages that have been acknowledged is to seek to a timestamp. To seek to a timestamp, you must first configure the subscription to retain acknowledged messages using retain-acked-messages. If retain-acked-messages is set, Pub/Sub retains acknowledged messages for 7 days.

You only need to do this step if you intend to seek to a timestamp, not to a snapshot.

https://cloud.google.com/pubsub/docs/replay-message

B and E, already tested at cloud console.

D & E. Make use of Seek feature of Pub/Sub which is available for pre-defined configuration.

DE.
The questions says "Delivered Messages", so this are a acknowledged messages, this can only do the D. retain-acked-messages flag.
Snapshot only save the unacknowledged messages, so, you miss the acknowledged to make a reprocess.

And Letter E is mandatory to comeback in time.

The only way to ensure that the updated pipeline reprocesses the previous two days of delivered Pub/Sub messages (current) is to use B&E.... for future D&E can work.

Since E is seek to a timestamp only one to replay, I think D is correct answer and not B

D and E

Please seem to not understand that the retain_acked_messages parameter needs to be enabled for any snapshot or seeking functionality to work.

After reading https://cloud.google.com/pubsub/docs/replay-overview#snapshot_overview carefully, B and E are correct.

First, read this document: https://cloud.google.com/pubsub/docs/replay-overview.

Key Points:

Seek to a Snapshot: Reprocesses only unacknowledged messages.
Seek to a Timestamp: Reprocesses all messages (acknowledged and unacknowledged) after that time.
Since the question asks for delivering all messages, option E is correct, as it includes both acknowledged and unacknowledged messages.

Regarding Option D: Configuring a subscription with the retain_acked_messages property allows replaying previously acknowledged messages retained for up to 31 days. This satisfies the requirement to deliver all messages and retains them longer than the mentioned 2 days.

B&D - based on Vertex AI feedback

BE
B. Use Pub/Sub Snapshot capture two days before the deployment.

Pub/Sub Snapshot: Creating a snapshot captures the state of the subscription at a specific point in time. You can then seek to this snapshot to replay messages from that point onwards.
By capturing a snapshot two days before the deployment, you can ensure that your pipeline reprocesses messages from the past two days.
E. Use Pub/Sub Seek with a timestamp.

Pub/Sub Seek: This feature allows you to reset the subscription to a specific timestamp. Messages published to the topic after this timestamp are re-delivered.
By seeking to the timestamp from two days ago, you can instruct Pub/Sub to start re-delivering messages from that point in time

D. Use the Pub/Sub subscription retain-acked-messages flag.
E. Use Pub/Sub Seek with a timestamp.

E for sure, you need to seek from a timestamp.
To accomplish to that you need to "Set the retain-acked-messages flag to true for the subscription."

From google documentation:

"Note: To seek to a previous time point, your subscription must be configured to retain acknowledged messages. You can change this setting by clicking Edit on the subscription details page, and checking the box for Retain acknowledged messages."

https://cloud.google.com/pubsub/docs/replay-message

D and E,
https://cloud.google.com/pubsub/docs/replay-message

B and E: The seek feature extends subscriber capabilities by allowing you to alter the acknowledgement state of messages in bulk. For example, you can replay previously acknowledged messages or purge messages in bulk. In addition, you can copy the acknowledgement state of one subscription to another by using seek in combination with a snapshot. Source: https://cloud.google.com/pubsub/docs/replay-overview

BE
https://cloud.google.com/pubsub/docs/replay-overview

Option D and E

Ans: A
B: Wrong eVM wont solve the problem of larger storage prices.
C: May be, but nothing mentioned in terms of what to tune in the question, also this is like-for-like migration so tuning may not be part of the migration.
D: Again, this is like-for-like so need to define what is hot data and which is cold data, also persistent disk costlier than cloud storage.

Ans A
Best Practice: Use Cloud Storage as the primary storage layer for Dataproc, with minimal Persistent Disk only for temporary processing and shuffle data. This is the cloud-native architecture Google recommends.

First I selected D, it just made sense right? It's faster to access data on the persistent disk. But then there are a couple of things to take into account:
- The question wants you to minimise cost. You only do this 100% on "A", "D" is still costing you a bit more.
- Performance, speed of access... is never mentioned. Common sense? Yes, but no one is asking you about that.
- In general Google Cloud's approach to Big Data architecture is to separate compute and storage, because of all the improvements and resilience GCS has.
- "D" is not very direct, has additional effort, and you could consider that cost.
- People mention the "like for like" section of the question. I'd argue like for like is both having a persistent disk, or having all data in one point. I'd not really consider it a point.

So all in all, A. It's just simpler and cheaper. Also, Gemini said so ;)

A like-for-like migration to Cloud Dataproc that replicates on-premises Hadoop would require each node to have 50 TB of persistent disk, which is costly. Instead, you can minimize storage costs by leveraging Google Cloud Storage (GCS). Cloud Dataproc seamlessly integrates with GCS through the Hadoop connector, allowing you to store your data cost-effectively in Cloud Storage and run ephemeral clusters that read data directly from GCS. This approach eliminates the need for each node to carry 50 TB of expensive persistent disk storage while still supporting your Hadoop workload.

. Migrate some of the cold data into Google Cloud Storage, and keep only the hot data in Persistent Disk.
Google Cloud Storage (GCS) is a cost-effective alternative to Persistent Disk for storing less frequently accessed ("cold") data.
Hot data that requires fast access can remain on Persistent Disk, reducing storage costs while maintaining performance.
Cloud Dataproc supports HDFS-to-GCS integration, allowing Hadoop jobs to access data in GCS seamlessly.D

You are most of the people looking at like for like migration would require 50TB persistent storage but missing to look at CIO concern about cost of block storage...considering CIO concern the option here is cloud storage... moreover that is recommended as well ..

Option A: Put the data into Google Cloud Storage.

This is the best option. Google Cloud Dataproc is designed to work well with Google Cloud Storage. Using GCS instead of Persistent Disk can save money, and GCS offers advantages such as higher durability and the ability to share data across multiple clusters.

I have seen this question in other places and I believe that you store the older data in Cloud Storage and retain processing data in persistent disk. D

Answer: D
Question: A like-for- like migration of the cluster would require 50 TB of Google Persistent Disk per node.
which means Persistent is still required.

the question is talking about block storage , GCS is object storage !

GCS is cost-effective and also Google's recommendation!

Minimize cost then GCS

A - is the right answer.

A - dataproc - storage - cost effective is cloud storage

Cloud Storage

Cloud Storage is google recommended one

Vote for 'A"

The Answer should be D. The custom endpoint is not acknowledging the message, that is the reason for Pub/Sub to send the message again and again. Not B.

D : Doubt should be only between B & D. But B is not possible because if SSL is expired then endpoint URL will not receive any messages forget about duplicates. So It should be D for Duplicates.

D. Your custom endpoint is not acknowledging messages within the acknowledgement deadline
Why This Causes Duplicate Messages
How Pub/Sub Delivery Works:

Pub/Sub delivers a message to your HTTPS endpoint
Your endpoint must return an HTTP success status (200, 201, 202, or 204) within the acknowledgement deadline (default is 10 seconds)
If Pub/Sub doesn't receive acknowledgement in time, it assumes delivery failed
Pub/Sub redelivers the message, causing duplicates

D
Why are there too many duplicate messages?

Pub/Sub guarantees at-least-once message delivery, which means that occasional duplicates are to be expected. However, a high rate of duplicates may indicate that the client is not acknowledging messages within the configured ack_deadline_seconds, and Pub/Sub is retrying the message delivery. This can be observed in the monitoring metrics pubsub.googleapis.com/subscription/pull_ack_message_operation_count for pull subscriptions, and pubsub.googleapis.com/subscription/push_request_count for push subscriptions. Look for elevated expired or webhook_timeout values in the /response_code. This is particularly likely if there are many small messages, since Pub/Sub may batch messages internally and a partially acknowledged batch will be fully redelivered.

D is correct
As per google docs- When you do not acknowledge a message before its acknowledgement deadline has expired, Pub/Sub resends the message. As a result, Pub/Sub can send duplicate messages. Use Google Cloud's operations suite to monitor acknowledge operations with the expired response code to detect this condition

The correct answer is D. Look for the link
https://cloud.google.com/pubsub/docs/faq

Why are there too many duplicate messages?
Pub/Sub guarantees at-least-once message delivery, which means that occasional duplicates are to be expected. However, a high rate of duplicates may indicate that the client is not acknowledging messages within the configured ack_deadline_seconds, and Pub/Sub is retrying the message delivery. This can be observed in the monitoring metrics pubsub.googleapis.com/subscription/pull_ack_message_operation_count for pull subscriptions, and pubsub.googleapis.com/subscription/push_request_count for push subscriptions. Look for elevated expired or webhook_timeout values in the /response_code. This is particularly likely if there are many small messages, since Pub/Sub may batch messages internally and a partially acknowledged batch will be fully redelivered.

Another possibility is that the subscriber is not acknowledging some messages because the code path processing those specific messages fails, and the Acknowledge call is never made; or the push endpoint never responds or responds with an error.

D as the Cloud Pub/Sub will deliver duplicate messages only if there has been no acknowledgement from the subscriber.
Refer GCP documentation - Cloud Pub/Sub FAQs - Duplicates:
https://cloud.google.com/pubsub/docs/faq#duplicates
Why are there too many duplicate messages?
Cloud Pub/Sub guarantees at-least-once message delivery, which means that occasional duplicates are to be expected. However, a high rate of duplicates may indicate that the client is not acknowledging messages within the configured ack_deadline_seconds, and Cloud Pub/Sub is retrying the message delivery.

If the answer had been B the messages had not reached the HTTP page since the push subscription requires an SSL certificate (https://cloud.google.com/pubsub/docs/push#:~:text=Endpoint%20URL%20(required)), so the answer must be D.

I think it is due to a bad response due to overcharge in the HTTPS page, which is sending a non correct status code (https://cloud.google.com/pubsub/docs/push#:~:text=VPC%20Service%20Controls.-,Receive%20messages,-When%20Pub/Sub)

In Google Cloud Pub/Sub, when you use a push subscription, messages are delivered to the specified endpoint (in this case, your custom HTTPS endpoint). The acknowledgment deadline is the time given to the endpoint to acknowledge that it has received and processed the message. If the acknowledgment is not received within the deadline, Pub/Sub may consider the message as unacknowledged and may attempt redelivery, which can lead to duplicate messages.

You should ensure that your custom HTTPS endpoint acknowledges messages within the acknowledgment deadline to prevent duplicate messages from being sent. Additionally, it's essential to handle messages in an idempotent way, so even if duplicates do occur, the action taken by your endpoint doesn't have unintended consequences.

After re-reading the question it seem to me that it is asking for a root cause. It is possible that the most common cause of this symptom is and expired certificate. Once expired duplicates would be received for every message.

Agree with previous explanations regarding validity of D

D - If a message has not been acknowledged within its acknowledgement deadline, Dataflow attempts to maintain the lease on the message by repeatedly extending the acknowledgement deadline to prevent redelivery from Pub/Sub. However this is best effort and there is a possibility that messages may be redelivered. This can be monitored using metrics listed here. https://cloud.google.com/blog/products/data-analytics/handling-duplicate-data-in-streaming-pipeline-using-pubsub-dataflow

D should be the answer. If acknowlegement is not received back to pub/sub , pub/sub may resend meassages.

The correct answer is:
D. Your custom endpoint is not acknowledging messages within the acknowledgement deadline.

if there were an out of date certificate then nothing would get through. D

It should be D
https://cloud.google.com/pubsub/docs/troubleshooting#dupes

The correct answer is D, because it's how Pub/Sub works.
Documentation here: https://cloud.google.com/pubsub/docs/troubleshooting#dupes

It should B, Storage Write API has "3 GB per second throughput in multi-regions; 300 MB per second in regions"

- BigQuery Storage Write API: This API is designed for high-throughput, low-latency writing of data into BigQuery. It also provides tools to prevent data duplication, which is essential for exactly-once delivery semantics.
- Regional Table: Choosing a regional location for the BigQuery table could potentially provide better performance and lower latency, as it would be closer to the Dataflow job if they are in the same region.

BigQuery is Regional component and not Multiregional. Hence Option:A

API BigQuery Storage Write with region because lower latency.

To achieve high-scale, exactly-once ingestion in BigQuery:

API: Always use the Storage Write API.

Mode: Use Committed Streams (for exactly-once) or the Default Stream (for at-least-once).

Scalability: Monitor the "Bytes per second" quota. If you are operating at "Gbps" levels, multiregional locations are the architectural standard due to higher throughput limits.

Why A is correct:
BigQuery Storage Write API:

Supports exactly-once semantics (deduplication via stream offsets).

Designed for high-throughput streaming ingestion (scales much better than the legacy Streaming Insert API).

Can handle your throughput requirement of 1.5 GB/s.

Regional table:

Required for exactly-once delivery guarantees with the Storage Write API.

3 GB per second throughput in multi-regions;
300 MB per second in regions

Go With (B) Not (A),
Max throughput for regional currently only 300 MB/s as for the dcos

Honestly, I think exam topic should do better job. the answers only make more mislead.

You need a write latency of 1.5 GBs per second. Given the high throughput requirement, a regional BigQuery table (Option A) is generally preferred over a multi-regional table due to potentially lower write latency in multi-region. Simple.

As per GCP document multi-region meets the troughput requirement.

It’s A

A. Implement the BigQuery Storage Write API and guarantee that the target BigQuery table is regional.

Here's the breakdown:

Why Option A is Superior

Exactly-Once Delivery: The BigQuery Storage Write API intrinsically supports exactly-once delivery using stream offsets. This guarantees that each message is written to BigQuery exactly one time, even in the case of retries due to the lack of native exactly-once support in your message bus.

High Throughput: The Storage Write API is optimized for high-throughput scenarios. It can handle the expected ingestion throughput of 1.5 GB per second.

Regional Tables: Using a regional BigQuery table aligns with best practices when utilizing the Storage Write API, as it helps to minimize latency and reduce potential cross-region communication costs.

- BigQuery Storage Write API: This API is designed for high-throughput, low-latency writing of data into BigQuery. It also provides tools to prevent data duplication, which is essential for exactly-once delivery semantics.
- The multiregional table ensures that your data is highly available and can be streamed into BigQuery across multiple regions. It is better suited for high-throughput and low-latency workloads, as it provides distributed write capabilities that can handle large data volumes, such as the 1.5 GB per second you expect to stream.

B is correct.
When aiming for exactly-once delivery in a Dataflow streaming job, the key is to use the BigQuery Storage Write API, as it provides the capability to handle large-scale data ingestion with the correct semantics, including exactly-once delivery.

3 GB per second throughput in multi-regions; 300 MB per second in regions
https://cloud.google.com/bigquery/quotas#write-api-limits

streamed into BigQuery with exactly-once delivery semantics >>> Storage Write API

ingestion throughput into BigQuery to be about 1.5 GB per second >>> multiregional (check throughput rate here >>> https://cloud.google.com/bigquery/quotas#write-api-limits)

Answer: D. The only that describe a way to resolve the problem, with buffering the data.

(the question is possible old, the best approach would be Pub/Sub + Dataflow Streaming + Bigquery for streaming data instead near-real time)

B. Streams data into BigQuery one record at a time without needing to run a load job: https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/insertAll
Instead of using a job to load data into BigQuery, you can choose to stream your data into BigQuery one record at a time by using the tabledata.insertAll method. This approach enables querying data without the delay of running a load job:
https://cloud.google.com/bigquery/streaming-data-into-bigquery
The BigQuery Storage Write API is a unified data-ingestion API for BigQuery. It combines the functionality of streaming ingestion and batch loading into a single high-performance API. You can use the Storage Write API to stream records into BigQuery that become available for query as they are written, or to batch process an arbitrarily large number of records and commit them in a single atomic operation.
Committed mode. Records are available for reading immediately as you write them to the stream. Use this mode for streaming workloads that need minimal read latency.
https://cloud.google.com/bigquery/docs/write-api

Option A Works:

Batch loading (even frequent batches like every 2 minutes) provides strong consistency
Once a batch load completes, the data is immediately and fully available for queries

A.
Loading data in batches (using batch loads instead of streaming inserts) ensures strong consistency in BigQuery.
Batch loads are available for querying immediately after the load job completes.
This approach is recommended when you need consistent query results right after data ingestion.

Answer D (which is most voted) is a workaround, not a solution

Why this is best:
Instead of inserting each message individually using streaming (which has consistency delay), you batch the data and load it using BATCH LOADS, which are strongly consistent once complete.
This reduces the chance of missing in-flight data in queries and avoids unnecessary complexity.
Loading in intervals (e.g. every 2 minutes) also helps control costs and improves query reliability.

not D because we want data in report in near real-time.

To address the issue of strong consistency and ensure your reports do not miss in-flight data after streaming inserts, you should re-write the application to load accumulated data every 2 minutes (option A).

Here's why:

By accumulating and loading data in 2-minute intervals, you can balance between real-time data processing and ensuring data consistency.

This approach allows you to process the data in manageable batches, reducing the likelihood of inconsistencies that might occur with individual streaming inserts.

It maintains a near real-time analysis capability while allowing enough time for all in-flight data to be captured and accurately represented in your reports.

This adjustment should help improve the reliability of your data analysis and reporting.

Accumulating data and loading it periodically (e.g., every 2 minutes) via batch inserts ensures strong consistency for queries. Batch loads in BigQuery allow you to avoid the latency issues inherent to streaming inserts and guarantee data availability for queries.

For maintaining data consistency while handling high throughput streaming inserts and subsequent aggregations in Google BigQuery, the best approach is to re-write the application to load accumulated data every 2 minutes.

"D" seems to use the typical approximate terminology of a wrong answer. "estimate the time" (how do you do that? do you do that over different times of the day?) and "wait twice as long" (who tells you that there are not a lot of cases when lag is twice as long?). Instead, "A" seems good. You don't need to show the exact results, but an approximation thereof, but you still want consistency. So an aggregation of the data every 2 minutes is a viable thing.

D is correct. The problem requirement is doing analytics on real-time data. You cannot do batch processing because the business requires it to be real-time even if it makes your job simpler, so B is incorrect. Other options are not streaming.

There are assumptions over the quality of data acceptable. If slight variations of the analytics against actual can be accepted, then D would be a good choice.

Many people chose B, but this also requires some form of waiting for the late data to arrive.

I think a combination of D and B can be applied, but for an intial fix, delaying the aggregation queries with D seems to make more sense. If the variance is small and the some late data leakage is acceptable, and we can remain as D.

If problems arise, we can always proceed to attempt B

The streaming mode may be in pending mode or buffered mode where the streaming data is not immediately available before committing or flushing. Thus, we need to wait before the data will be available. Or else we need to switch to commited mode (which is not present in the choices).

Answer: D
What to learn or look for
1. In-Flight data = (Real Time data, i.e still in streaming pipeline and not landed in BigQuery)
2. Dataflow (assume in best case) streaming pipeline is running to send data to Bigquery.
Why not option B: change streaming to batch upload is not business requirement, we have to stuck to streaming and real time analysis.

Option D: make bigquery run after waiting for sometime (twice here), How will you do it?
- there is not setting in BQ to do it, right!. So, adjust it in your pipeline (dataflow)
- For example, add Fixed window, and you want to execute aggregation query after 2 min.
Code
```pipeline.apply(...)
.apply(Window.<TableRow>into(FixedWindows.of(Duration.standardMinutes(2))))
.apply(BigQueryIO.writeTableRows()
.to("my_dataset.my_table")
```

Answer: D
I agree with the first part of the D answer, but for the second part, I don't know how they came about the 2 mins, is it from a calculation?

A. Re-write the application to load accumulated data every 2 minutes.

By accumulating data and performing a batch load every 2 minutes, you can reduce the potential inconsistency caused by streaming inserts. While this introduces a slight delay, it provides a more consistent approach than streaming each individual message. This method can still meet the near real-time requirement, and the slight delay is often acceptable in scenarios where data consistency is paramount.

BBBBB is the only option

Is this really a question that could appear in Google Cloud Professional Data Engineer Exam? What does it have to do with Google Cloud? I would use DataProc no?

A, C and D - all are valid answers. You can do checkpointing, splitting pipelines and merging pipelines with all three options.

The most appropriate method to write complex ETL pipelines requiring explicit checkpointing and splitting pipelines directly on an Apache Hadoop cluster is Mapreduce. Java or Python both are supported but Java is primary language and so Option C

Réponse : C
Explication :
La bonne réponse ici est C. Java using MapReduce.
Explication rapide :
Tu parles de besoin de "checkpointing" et "splitting pipelines".
PigLatin (Pig) et HiveQL (Hive) sont des langages déclaratifs (de plus haut niveau), pas faits pour un contrôle précis sur la manière dont les jobs se séquencent, checkpointent ou s'articulent.
MapReduce en Java te donne un contrôle total sur :
L'ordonnancement précis.
Le checkpointing manuel.
Le découpage et le chaînage des étapes.
Python avec MapReduce est possible mais beaucoup moins natif ; Hadoop est conçu principalement pour Java MapReduce, donc en Python ce serait plus compliqué, plus fragile et moins performant.
Résumé rapide pour que tu t'en souviennes :👉 Besoin de contrôle précis sur la logique ETL complexe = MapReduce en Java.

MapReduce in Java (C) allows more control for checkpointing and splitting.

Writing your pipeline in Java using MapReduce allows you to implement these custom controls and fine-tune the execution, ensuring robust and manageable ETL processes on your Hadoop cluster

Pig Latin supports both splitting pipelines and checkpointing, allowing users to create complex data processing workflows with the ability to restart from specific points in the pipeline if necessary.

I would go to A.
C, D are similar. So both are excluded. B, Hive is actually a data warehouse system. I don't use Apache Pig. But, BCD are wrong. Then A should be correct.

A as others have said

Comment content is too short

PigLatin is the correct answer, however... the last release was 6 years ago and has lots of bugs.

This answer depends which language you are comfortable with.
Hadoop is your framework, where mapReduce is your Native programming model in JAVA, which is designed to scale, parallel processing, restart pipeline from any checkpoint etc. , So if you are comfortable with JAVA, you can customize your checkpoint at lowlevel in better way. otherwise, choose PIG which is another programming concept run over JAVA but then you need to learn this also, if not choose python as it can be deployed with hadoop because hadoop has been making updates for python clients regularly.
Option C: is the best one.

C. Java using MapReduce or D. Python using MapReduce

Apache Hadoop is a distributed computing framework that allows you to process large datasets using the MapReduce programming model. There are several options for writing ETL pipelines to run on a Hadoop cluster, but the most common are using Java or Python with the MapReduce programming model.

Description: Pig is scripting language which can be used for checkpointing and splitting pipelines

Why not D?

PigLatin supports checkpoints

Answer: A

C.
This is a tricky one. The issue here is the unreliable connection between data collection and data processing infrastructure, and to resolve it in a cost-effective manner. However, it also mentions that the company is using leased lines. I think replacing the leased lines with Cloud InterConnect would solve the problem, and hopefully not be an added expense.
https://cloud.google.com/interconnect/docs/concepts/overview

B. Have the data acquisition devices publish data to Cloud Pub/Sub. This would provide a reliable messaging service for your event data, allowing you to ingest and process your data in a timely manner, regardless of the reliability of the leased lines. Cloud Pub/Sub also offers automatic retries and fault-tolerance, which would further improve the reliability of your event delivery. Additionally, using Cloud Pub/Sub would allow you to easily scale up or down your event processing infrastructure as needed, which would help to minimize costs.

Option A: Deploy small Kafka clusters in your data centers
• Kafka acts as a local buffer for events.
• Sensors write to the local Kafka cluster, ensuring no data loss even if connectivity to the central processing infrastructure is slow or unreliable.
• Once connectivity is available, Kafka can replicate or forward events to the central processing pipeline.
• This is cost-effective compared to provisioning dedicated interconnects or streaming directly to cloud services with unreliable lines.
• Kafka is designed for high-throughput, low-latency buffering, making it ideal here.

The data acquisition devices are spread across the globe. How many cloud interconnects would this require? Option C is definitely not cost effective. Option B seems to make sense as it suggests publishing from the acquisition device to Pub/Sub and not from the collection infrastructure. The problem clearly stated the connection from collection infrastructure to the processing infrastructure was unreliable.

Option B: Have the data acquisition devices publish data to Cloud Pub/Sub.

Rationale:

Managed Service: Cloud Pub/Sub is a fully managed service, reducing the operational overhead compared to managing Kafka clusters.
Reliability and Scalability: Cloud Pub/Sub can handle high volumes of data with low latency and provides built-in mechanisms for reliable message delivery, even in the face of intermittent connectivity.
Cost-Effective: Cloud Pub/Sub offers a pay-as-you-go pricing model, which can be more cost-effective than setting up and maintaining dedicated network infrastructure like Cloud Interconnect.
Global Availability: Cloud Pub/Sub is available globally and can handle data from multiple regions efficiently.

Even with Cloud Pub/Sub, unpredictable latency or delays could still occur due to the unreliable leased lines connecting your event collection infrastructure and event processing infrastructure. While Cloud Pub/Sub offers reliable message delivery within its own network, the handoff to your processing infrastructure is still dependent on the leased lines.
Replacing leased lines with Cloud Interconnect could potentially resolve the overall issue of unpredictable latency in event processing pipeline but it could be unnecessary expense provided data centers distributed world wide.
Cloud Pub/Sub along with other optimization techniques like Cloud VPN or edge computing might be sufficient.

I don't know why B is the most voted. The issue here is unreliable connectivity and C is the perfect use-case for that

its says with unpredictable latency and here no need to worry about connection
So B is the right one

The question is misleading. But should be C since it addresses the unpredictablility and latency directly.

Best answer is A, By using Kafka, you can buffer the events in the data centers until a reliable connection is established with the event processing infrastructure.
But go with B, its google asking :P

Are they talking about GCP in this question?
Where is the event processing infrastructure?

Answer A, might be correct!

pub/sub is region is a global service
It's important to note that the term "global" in this context refers to the geographical scope of the service

As usual the answer is hidden somewhere in the Google Cloud Blog:
"In the case of our automotive company, the data is already stored and processed in local data centers in different regions. This happens by streaming all sensor data from the cars via MQTT to local Kafka Clusters that leverage Confluent’s MQTT Proxy."
"This integration from devices to a local Kafka cluster typically is its own standalone project, because you need to handle IoT-specific challenges like constrained devices and unreliable networks."

🔗 https://cloud.google.com/blog/products/ai-machine-learning/enabling-connected-transformation-with-apache-kafka-and-tensorflow-on-google-cloud-platform

yeah, changing whole architecture arround the world for the use of pub/sub is so much more cost efficient than Cloud Interconnect (which is like 3k$)..

It's C.

Cloud Pub/Sub, it supports batch & streaming , push and pull capabilities
Answer B

It has to be B.

Feels like everyone is wrong.

A. Deploy small Kafka clusters in your data centers to buffer events.
- Silly in a GCP cloudnative context, plus they have messaging infra anyway
B. Have the data acquisition devices publish data to Cloud Pub/Sub.
- They have messaging infra, so why? Unless they want to replace, it, but that doesn't change the issue
C. Establish a Cloud Interconnect between all remote data centers and Google.
- Wrong, because Interconnect is basically a leased line. There must be some telecoms issue with it, which we can assume is unresolvable e.g. long distance remote locations and sometimes water ingress, and the telco can't justify sorting it yet, or is slow to, or something. Leased lines usually don't come with awful internet connectivity, so sound physical connectivity issue. Sure, an Interconnect is better, more direct, but a leased line should be bullet proof.
D. Write a Cloud Dataflow pipeline that aggregates all data in session windows.
- The only way to address dodgy/delayed data delivery

Correct Option : A
Explanation:-This option is correct as the key requirement is not to lose
the data, the Dataflow pipeline can be stopped using the Drain option.
Drain options would cause Dataflow to stop any new processing, but would
also allow the existing processing to complete

Answer is D.
* A and B are not possible, since the new job is not compatible.
* C might lead to lost data.
* D might lead to data being processed twice, but no data will be lost.
Better would usually be to drain and start a new pipeline.

B is correct: Update the current pipeline and provide the transform mapping JSON object.

This is the Google Cloud recommended way to transition between streaming pipeline versions without data loss:
1. Create a new subscription on the same Pub/Sub topic.
2. New subscription starts receiving messages from the moment of creation.
3. Create a new pipeline with this new subscription.
4. After validation, cancel the old pipeline and subscribe if necessary.

Please refer the google doc link below,
https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#drain

Create a new pipeline that has a new Cloud Pub/Sub subscription and cancel the old pipeline

D. Create a new pipeline that has a new Cloud Pub/Sub subscription and cancel the old pipeline:

By creating a new subscription, the new pipeline will consume messages independently of the old pipeline. This ensures no data is lost as messages published to Pub/Sub are delivered to all subscriptions.
Once the new pipeline is verified to be running as expected, the old pipeline can be safely canceled.

Why option D is better for this case - In this scenario: the pipeline is incompatible with the old one; running the two pipelines concurrently ensures no data loss and allows for easier debugging of the new pipeline; a new subscription ensures the old pipeline can finish processing its messages while the new pipeline starts fresh

Draining a pipeline stops it in an orderly manner but does not address incompatibility issues. Once the pipeline is drained, no more data is processed, and the new pipeline starts fresh. This can lead to data loss if there are messages in Pub/Sub that the drained pipeline didn't process.

Option: C
using draining will stop the subscription totally while allowing the existing data to complete processing. While the pipeline is stopped, will lose streaming data. The best option is to create a new pipeline that is connected to the same subscription, then we can apply drain to the old pipeline and end it. That way we will capture all the streaming data.

Drain flag: This flag allows the pipeline to finish processing all existing data in the Pub/Sub subscription before shutting down. This ensures no data is lost during the update.
Current pipeline: Updating the current pipeline minimizes disruption and avoids setting up entirely new infrastructure.
Incompatible changes: Even with incompatible changes, the drain flag ensures existing data is processed correctly.

Same Cloud Pub/Sub Subscription: By using the same Cloud Pub/Sub subscription for the new pipeline, you ensure that no messages are lost during the transition. Pub/Sub manages message delivery, ensuring that unacknowledged messages (those that haven't been processed by your old pipeline) will be available for the new pipeline to process.

Creating a New Pipeline: Since the update makes the new pipeline incompatible with the current version, it's necessary to create a new pipeline. Attempting to update the current pipeline in place (options A and B) would not be feasible due to compatibility issues.

Cancel the Old Pipeline: Once the new pipeline is up and running and processing messages, you can safely cancel the old pipeline. This ensures a smooth transition with no data loss.

In order to make an update to a Google Cloud Dataflow streaming pipeline without losing any data, the recommended approach is:

A. Update the current pipeline and use the drain flag.

Explanation:

The drain flag is designed to allow the current pipeline to finish processing any remaining data before shutting down. This helps ensure that no data is lost during the update process.
By updating the current pipeline and using the drain flag, you allow the pipeline to complete its current processing before the update takes effect, minimizing the risk of data loss.
This approach is a safe way to transition from the old version to the new version without interrupting data processing.

I would vote for A because of the structure of the exam, but there are other options worth considering as well

My answer is C. Chatted with ChatGPT and narrowed down on this option. Let me know your thoughts on this perspective.
Option C - By using the existing subscription, you can ensure that the data flow remains uninterrupted, and there is no loss of data during the transition from the old pipeline to the new one.

Creating a new pipeline that uses the same Cloud Pub/Sub subscription allows for a seamless transition without any interruptions to the data flow. This approach ensures that the new pipeline can continue to consume data from the same subscription as the old pipeline, thereby maintaining data continuity throughout the update process.

Correct Option : A
Explanation:-This option is correct as the key requirement is not to lose
the data, the Dataflow pipeline can be stopped using the Drain option.

It should be B
Drain will stop the existing job only and it does not suffice the updated schema.
In order to bring updated schema into effect, updated JSON mapping need to be applied.

The answer is D. An ETL pipeline will be implemented for this scenario. Check out handling invalid inputs in cloud data flow

https://cloud.google.com/blog/products/gcp/handling-invalid-inputs-in-dataflow

ParDos . . . and don’ts: handling invalid inputs in Dataflow using Side Outputs as a “Dead Letter” file

Option A is incorrect because federated data sources do not provide any data validation or cleaning capabilities and you'll have to do it on the SQL query, which could slow down the performance.

Option B is incorrect because Stackdriver monitoring can only monitor the performance of the pipeline, but it can't handle corrupted or incorrectly formatted data.

Option C is incorrect because using gcloud CLI and setting max_bad_records to 0 will ignore the corrupted or incorrectly formatted data and continue the load process, this will lead to incorrect analysis.

Answer D. Run a Google Cloud Dataflow batch pipeline to import the data into BigQuery, and push errors to another dead-letter table for analysis.

D. Run a Google Cloud Dataflow batch pipeline to import the data into BigQuery, and push errors to another dead-letter table for analysis.

"you want to keep the data"

You should transform the raw data by eliminate the error before analyse.

Google Cloud Dataflow allows you to create a data pipeline that can preprocess and transform data before loading it into BigQuery. This approach will enable you to handle problematic rows, push them to a dead-letter table for later analysis, and load the valid data into BigQuery.

Option A (using federated data sources and checking data in the SQL query) can be used but doesn't directly address the issue of handling corrupted or incorrectly formatted rows.

Options B and C are not the best choices for handling data quality and error issues. Enabling monitoring and setting max_bad_records to 0 in BigQuery may help identify errors but won't store the problematic rows for further analysis, and it might prevent loading any data with issues, which may not be ideal.

D. Run a Google Cloud Dataflow batch pipeline to import the data into BigQuery, and push errors to another dead-letter table for analysis.

By running a Cloud Dataflow pipeline to import the data, you can perform data validation, cleaning and transformation before it gets loaded into BigQuery. Dataflow allows you to handle corrupted or incorrectly formatted rows by pushing them to another dead-letter table for analysis. This way, you can ensure that only clean and correctly formatted data is loaded into BigQuery for analysis.

All other options only alert or error out bad data. As the question requires, option D sends bad data to the dead letter table for further analysis while valid data is loaded to the table

Agreed: D

D because you need Transform the data

D. The question is asking pipeline, then let’s build a pipeline.

Agreed: D

The correct answer is D

Correct - D (as we need to create Pipeline) which possible via 'D'

Looks like D, with C you will not import anything, stackdriver alerts will not help you with this and with federated resources you won’t know what happened with those bad records. D is the most complete one.
https://cloud.google.com/blog/products/gcp/handling-invalid-inputs-in-dataflow

Ans: D

D seems to be correct. explained here how combined wth Pub/Sub, this can be achieved. https://cloud.google.com/pubsub/docs/handling-failures

- Graceful Data Transition: Draining the old pipeline ensures it processes all existing data in its buffers and watermarks before shutting down, preventing data loss or inconsistencies.
- Minimal Latency Increase: The latency increase will be limited to the amount of time it takes to drain the old pipeline, typically within the acceptable 10-minute threshold.

I don't think C is correct, as it will immediately fire the window:
"Draining can result in partially filled windows. In that case, if you restart the drained pipeline, the same window might fire a second time, which can cause issues with your data. "
https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#effects

Maybe "A" means launching a replacement job?
https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline#Launching

Draining can result in partially filled windows. In that case, if you restart the drained pipeline, the same window might fire a second time, which can cause issues with your data. For example, in the following scenario, files might have conflicting names, and data might be overwritten.
https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline

Drain (C)

Avoids in-flight loss, but can create inconsistent results due to early window closure.

This violates “no inconsistencies.”

Snapshot (B)

Captures state + offsets, lets new job continue without gaps or duplicates.

Meets “no data loss / no inconsistencies.”

Latency bump stays well under 10 minutes in practice.

A snapshot captures the exact ack state of the subscription. Starting the new pipeline from that snapshot guarantees no data loss and no duplicates, preserving window consistency even if your windowing/logic changed.
(C) Draining waits for in-flight windows/watermarks to complete; with a 5-minute window this can easily exceed the allowed +10 minutes latency and still risks a gap while you switch pipelines.

Why not B?
https://cloud.google.com/dataflow/docs/guides/upgrade-guide#stop-and-replace

There is requirement to avoid data loss.

https://cloud.google.com/dataflow/docs/guides/upgrade-guide#stop-and-replace
"To avoid data loss, in most cases, draining is the preferred action."

- Draining the old pipeline ensures that it finishes processing all in-flight data before stopping, which prevents data loss and inconsistencies.
- After draining, you can start the new pipeline, which will begin processing new data from where the old pipeline left off.
- This approach maintains a smooth transition between the old and new versions, minimizing latency increases and avoiding data gaps or overlaps.

==> Other options, such as updating, snapshotting, or canceling, might not provide the same level of consistency and could lead to data loss or increased latency beyond the acceptable 10-minute window. Draining is the safest method to ensure a seamless transition.

I would choose B as mentioned by Alizcert, a simple drain may cause problem
Dataflow snapshots save the state of a streaming pipeline, which lets you start a new version of your Dataflow job without losing state. Snapshots are useful for backup and recovery, testing and rolling back updates to streaming pipelines, and other similar scenarios.

C option

Option C, draining the old pipeline solves all requests

C. Drain the old pipeline, then start the new pipeline.

The Answer should be B. The Dataflow developer role will not provide access to the underlying data.

Not sure who it could be anything apart from D. if I put myself in developer shoes then without seeing the data how can I develop any logic let alone the complex one. and if I have access thru any means (ex. service account) then I can just print the logs and see the data in the logs anyway. So option D appears to be the only option.

A. Viewer role: Still exposes all resources and metadata in the project, too broad.

B. Dataflow Developer role: Lets them run pipelines on the real project (and potentially access private data).

C. Service account sharing: A security anti-pattern; violates identity/accountability.

D. Anonymized sample in a separate project: Safest and compliant option—consultant can code and test transformations without ever seeing sensitive data.

I think only D really covers user's privacy

You’re dealing with sensitive user data (possibly PII or other regulated content).
Principle of least privilege and data minimization are key.
External consultants should never be given access to production data unless absolutely necessary — and in this case, it's not.
Creating an anonymized or synthetic dataset lets them do their work (e.g., test code or build transformations) without compromising real user data.
Using a separate project also provides better audit boundaries and isolation.

D because the data is anonymized and that satisfies the privacy part of the question. It also isolates the consultant's work from your actual work. This is standard for privacy and the consultant's work will not impact your actual project.

logical answer for me

The answer cannot be B, because B is too retrictive, it can only create and manage dataflow jobs, but cannot view data. I acknowledge that is secure, but no consultant can do the job without seeing representative test data. D is the only one that provides enough to do the job, while still remaining totally private.

D cannot be the answer because the question clearly states the developer has to work in your project. Creating another project is not in scope and is a waste of time. Correct answer is B. Developer role has developer rights only, no view rights.

A. Grant the consultant the Viewer role on the project.
This role provides read-only access to all resources in the project, which could expose sensitive data to the consultant, violating privacy principles.

B. Grant the consultant the Cloud Dataflow Developer role on the project.
This role allows the consultant to create and manage Dataflow jobs but does not give them access the underlying data, it is not sufficient, the developer still needs data.

- C. Create a service account and allow the consultant to log on with it.
Allowing the consultant to log on with a service account could grant them access to sensitive data if the service account has broad permissions. This approach does not address the need to limit data exposure.

- D. Create an anonymized sample of the data for the consultant to work with in a different project.
- this fits the requrements

D
B is a good option to maintain privacy of sensitive data, but he also need some test data to validate the transformation logic right, so creating sample data and allow him to test in another project seems good.

Data flow data privacy rules cant allow the developer to see what the data, He/she just designs the pipelines and the flow as the interdependent tasks for the composer

B as the Dataflow developer role would help provide the third-party consultant access to create and work on the Dataflow pipeline. However, it does not provide access to view the data, thus maintaining user's privacy.
Refer GCP documentation - Dataflow roles:
https://cloud.google.com/dataflow/docs/concepts/access-control#roles
Option A is wrong as it would not allow the consultant to work on the pipeline.
Option C is wrong as the consultant cannot use the service account to login.
Option D is wrong as it does not enable collaboration.

C and A will not maintain user's privacy so out. B without data will be enough. D will give a good sample data, maintain privacy and the consultant will help creating the dataflow pipe for the project as requested. so D.

I follow the corresponding logic choosing between B and D:

Yes, with the Dataflow Developer role it is possible to execute and manipulate Dataflow jobs, but do we need to execute it? Based on my understanding we only need to ask for help to write it down. Is it possible without having access to test the data? I don't think so. At the same time, we need to perform an anonymization on it. So the answer D is more appropriate for me

By creating an anonymized sample of the data, you can provide the consultant with a realistic dataset that doesn't contain sensitive or private information. This way, the consultant can work on the project without direct access to sensitive data, reducing privacy risks.

Options A and B involve granting the consultant access to the project, which may expose sensitive data, even if they have limited permissions.

Option C involves creating a service account, but it doesn't address the need to anonymize the data or provide a separate, safe environment for the consultant to work with.

Option D provides a controlled environment that allows the consultant to work effectively while maintaining data privacy.

D. Create an anonymized sample of the data for the consultant to work within a different project.

By creating an anonymized sample of the data, you can provide the consultant with a realistic dataset that doesn't contain sensitive or private information. This way, the consultant can work on the project without direct access to sensitive data, reducing privacy risks.

Options A and B involve granting the consultant access to the project, which may expose sensitive data, even if they have limited permissions.

Option C involves creating a service account, but it doesn't address the need to anonymize the data or provide a separate, safe environment for the consultant to work with.

Option D provides a controlled environment that allows the consultant to work effectively while maintaining data privacy.

- Fusion optimization in Dataflow can lead to steps being "fused" together, which can sometimes hinder parallelization.
- Introducing a Reshuffle step can prevent fusion and force the distribution of work across more workers.
- This can be an effective way to improve parallelism and potentially trigger the autoscaler to increase the number of workers.

https://cloud.google.com/dataflow/docs/pipeline-lifecycle#prevent_fusion

Right fitting is for declaration, declaring the correct resources will not help. Reshuffling step is what can prevent fusion which can lead to unused workers.

Fusion occurs when multiple transformations are fused into a single stage, which can limit parallelism and hinder performance, especially in streaming pipelines. By introducing a Reshuffle step, you break fusion and allow for better parallelism.

https://cloud.google.com/dataflow/docs/guides/right-fitting

Problem is performnace and not using all workers properly, https://cloud.google.com/dataflow/docs/pipeline-lifecycle#fusion_optimization

D. Use Dataflow Prime, and enable Right Fitting to increase the worker resources.

I think it should be B because we have to use a combination of old and new test data as well as training data

B, as we need to train the data with new data, so that it will keep learning, and as well as used for test

You always need to retrain on such combination

Answer is B:
Fow new scenarios, we must need use old and new data for training, for continuos learning of patterns, etc.

Answer is B, need to take acount both data to train

We should continuously retrain existing data with latest data to balance the need to adapt to new changes while not overriding historical knowledge. This is called Continuous Retraining where the model is periodically updated with latest data so that recommendations remain accurate over time.

Continuously retrain the model on a combination of existing data and the new data.

From my point of view, we should take both datasets.

Option A is not recommended because retraining the model on just new data will cause the model to lose the information it has learned from the historical data.

Option C and D are not recommended because they are using the new data as test set and this approach will lead to a model that is overfitting and not generalize well to new users.

So answer is B

Answer is C. It is time sensitive data so latest data should be used for testing.
Reference: https://cloud.google.com/automl-tables/docs/prepare#ml-use

This approach allows the model to benefit from both the historical data (existing data) and the new data, ensuring that it adapts to changing preferences while retaining knowledge from the past. By combining both types of data, the model can learn to make recommendations that are up-to-date and relevant to users' evolving preferences.

train on old and new data

Option B is the right answer. Since the questions states the models needs to be updated since the clothing preference changes. Hence we need the new data to be utilized for training/ updating model.