Google Professional Data Engineer Governance and Security

- The most straightforward solution with minimal configuration overhead.
- By creating the "gdpr" tag template with public visibility, you ensure that all employees can search and find tables based on the "has_sensitive_data" field.
- Assigning the bigquery.dataViewer role to the HR group on tables with sensitive data ensures that only they can view the actual data in these tables.

"All employees must be able to do a simple search and find tables in the dataset that have either true or false in the “has_sensitive_data’ field." To be able to search for values in the tags you need the role roles/datacatalog.tagTemplateViewer. Meaning option D is correct.

This Guy Raasd is mostly correct with explanation thanks mate.

A - employees cannot use the tag
B - increases the configuration overhead
C - exactly what we need
D - unnecessary role assignment, the tag template is already visibile

While D works well, it is not obligated to give all employees the role of tagTemplateViewer, as it will give them the view permission for tag templates as well as the tags created by the template.
However, Tags are a type of business metadata. Adding tags to a data entry helps provide meaningful context to anyone who needs to use the asset.And public tags provide less strict access control for searching and viewing the tag as compared to private tags. Any user who has the required view permissions for a data entry can view all the public tags associated with it. View permissions for public tags are only required when you perform a search in Data Catalog using the tag: syntax or when you view an unattached tag template.

I'll go with raaad's answer

If you working with PII, We can't granted public access. So Private Visibility for the Tag Template its the best option.

Check it https://cloud.google.com/data-catalog/docs/tags-and-tag-templates

D. Create the “gdpr” tag template with public visibility. Assign the datacatalog.tagTemplateViewer role on this tag to the all employees group, and assign the bigquery.dataViewer role to the HR group on the tables that contain sensitive data.

C is too complex

Option B & E

B& E
https://cloud.google.com/bigquery/docs/column-level-security-intro

B & E. As per Google recommended security best practices

the correct options are:

A. Create two separate authorized datasets; one for the data analytics team and another for the consumer support team.
C. Replace the authorized dataset with an authorized view. Use row-level security and apply filter_expression to limit data access.

B & E
B - It will ensure they don't have access to secure columns
E- It will allow to enforce column level security
Ref - https://cloud.google.com/bigquery/docs/column-level-security-intro

Option B& E to me

A & B
The current setup is not effective because the data analytics team still has access to the sensitive columns despite using an authorized dataset and policy tags. This indicates that the policy tags are not being enforced properly, and the data analytics team members are able to view the tags and gain access to the sensitive data.
Separating the data into two distinct authorized datasets is a better approach because it isolates the sensitive data from the non-sensitive data. This prevents the data analytics team from accessing the sensitive columns directly, even if they have access to the authorized dataset for general customer data.
Additionally, revoking the Data Catalog Fine-Grained Reader role from the data analytics team members ensures that they cannot view or modify the policy tags. This limits their ability to bypass the access control imposed by the authorized dataset and policy tags.

And the second answer? One is option B and the other is option D maybe?

- The Data Catalog Fine-Grained Reader role allows users to read metadata that is restricted by policy tags.
- If members of the data analytics team have this role, they might bypass the restrictions set by policy tags.
- Ensuring they do not have this role will help enforce the restrictions intended by the policy tags.

Prevents data analytics team members from viewing sensitive data, even if it's tagged.
Restricts access to policy tags themselves, ensuring confidentiality of sensitive information.

- It utilizes Data Catalog's native support for both BigQuery datasets and Pub/Sub topics.
- For PostgreSQL tables running on a Compute Engine instance, you'd use Data Catalog APIs to create custom entries, as Data Catalog does not automatically discover external databases like PostgreSQL.

Data Catalog is the best choice. But for catalogging PostgreSQL it is better to use a connector when available, instead of using API.
https://cloud.google.com/data-catalog/docs/integrate-data-sources#integrate_unsupported_data_sources

Automatic Cataloging: Data Catalog (now part of Dataplex) has built-in integrations to automatically discover and register metadata from Google Cloud managed services like BigQuery and Pub/Sub. This requires little to no configuration.
Custom Connector for PostgreSQL: For the PostgreSQL instance running on Compute Engine, you'll need to develop a "custom connector." This typically involves writing a script or application that:
Connects to the PostgreSQL database.
Extracts schema information (tables, columns, types, etc.), potentially from information_schema.
Transforms this information into the format required by the Data Catalog API.
Uses the Data Catalog APIs to create or update entries for the PostgreSQL tables. This connector can be run on a schedule (e.g., using Cloud Scheduler and Cloud Functions) to keep the metadata in Data Catalog synchronized with the PostgreSQL instance.

This is the correct approach. Data Catalog provides automatic discovery and metadata extraction for both BigQuery datasets and Pub/Sub topics as native Google Cloud services. You'll only need to use the Data Catalog APIs to manually catalog the PostgreSQL tables running on Compute Engine, as this is a non-native data source. This maximizes automation while minimizing development effort.

Why C is Correct?
BigQuery datasets → ✅ Automatically cataloged in Data Catalog

Pub/Sub topics → ✅ Automatically cataloged in Data Catalog

PostgreSQL on Compute Engine → ❌ Not automatically cataloged

Requires a custom connector to extract metadata and push it to Data Catalog.

Option B (using Data Catalog APIs manually) is not enough because PostgreSQL metadata isn’t natively supported.

Why Not B?
Option B suggests using Data Catalog APIs manually for PostgreSQL.

However, Data Catalog does not natively support PostgreSQL metadata extraction.

You need a custom connector to first extract PostgreSQL schema information, then push it to Data Catalog.

This section explains it clearly: https://cloud.google.com/data-catalog/docs/integrate-data-sources#integrate_unsupported_data_sources.

This is C. To clarify some issues below with B, the links provided by supporters of B actually do say that it's preferable to use a community connector where available, and to only use the API when the case is genuinely not supported by community connectors.
In this case it's Postgresql, so it's supported, see here for full list: https://cloud.google.com/data-catalog/docs/integrate-data-sources#integrate_on-premises_data_sources

So this would be B if it was something like Q+ or some genuinely unsupported database, but postgres is supported for community connector.

Data Catalog automatically catalogs metadata from Google Cloud sources such as BigQuery, Vertex AI, Pub/Sub, Spanner, Bigtable, and more.

To catalog metadata from non-Google Cloud systems in your organization, you can use the following:

Community-contributed connectors to multiple popular on-premises data sources
Manually build on the Data Catalog APIs for custom entries

raaad mostly correct and we can check his description supporting his answer so we can go with it .Cheers mate

I’m voting for C because the documentation states that Postgres is a custom connector developed by the community.

BigQuery Datasets and Pub/Sub Topics: Google Data Catalog can automatically catalog metadata from BigQuery and Pub/Sub, making it easy to discover and manage these data assets without additional development effort.

PostgreSQL Tables: While Data Catalog does not have built-in connectors for PostgreSQL, you can use the Data Catalog APIs to manually catalog the PostgreSQL tables. This requires some custom development but is manageable compared to creating custom connectors for everything.

B. Use Data Catalog to automatically catalog BigQuery datasets and Pub/Sub topics. Use Data Catalog APIs to manually catalog PostgreSQL tables.

Option B leverages Data Catalog to automatically catalog BigQuery datasets and Pub/Sub topics, which streamlines the process and reduces manual effort. Using Data Catalog APIs to manually catalog PostgreSQL tables ensures consistency across all data assets while minimizing development and configuration efforts.

I vote for c as per Integrate on-premises data sources
To integrate on-premises data sources, you can use the corresponding Python connectors contributed by the community:

under the link

https://cloud.google.com/data-catalog/docs/integrate-data-sources

In the opction C, the expression "Use custom connectors to manually catalog PostgreSQL tables." is refering to the use case of Google when you want to use "Community-contributed connectors to multiple popular on-premises data sources". As you can see, this connectors are for ON-PREMISSES data sources ONLY. In this case the Postgres is in a VM in the cloud. Thus, the option correct is B.

option B, there's no need to build a custom connector now, postgreSQL is now supported
https://github.com/GoogleCloudPlatform/datacatalog-connectors-rdbms/tree/master/google-datacatalog-postgresql-connector

Use Data Catalog to automatically catalog BigQuery datasets and Pub/Sub topics. Use Data Catalog APIs to manually catalog PostgreSQL tables.

Should be A. Curated zone need Parquet, Avro, ORC format not CSV or JSON. Check the ref - https://cloud.google.com/dataplex/docs/add-zone#curated-zones

- Auto-Discovery Feature: Dataplex has an auto-discovery feature that, when enabled, automatically discovers and catalogs data assets within a zone.
- Appropriate for Both Raw and Curated Zones: This feature is applicable to both raw and curated zones, and it should be tailored to the specific data governance and cataloging needs of the organization.

Still if you go with "A" you need to do "B". the question is not about best practice.

We can store JSON and CSV in the curated zone if those files represent curated data. Usually we store JSON/CSV files in the raw zone if they are straight from source. But nowhere in the question is any of that detail mentioned. So I think the correct answer is :
B - Dataplex automatically discovers and catalogs data in the zones only if auto-discovery is enabled for the zone or asset. In this scenario - JSON and CSV files are being uploaded to a curated zone, which is fine. But if files are not being discovered, it's likely because auto-discovery is not enabled for that zone.

Answer is A
Data Format supported: Data in curated zones is typically columnar, Hive-partitioned, and stored in formats like Parquet, Avro, or ORC
Restrictions: Dataplex does NOT allow users to create CSV files within a "curated zone

- Raw zones store structured data, semi-structured data such as CSV files and JSON files, and unstructured data in any format from external sources. Raw zones are useful for staging raw data before performing any transformations. Data can be stored in Cloud Storage buckets or BigQuery datasets.
- Curated Zones do not support JSON / CSV

Raw zones store structured data, semi-structured data such as CSV files and JSON files, and unstructured data in any format from external sources. Curated zones store structured data. Data can be stored in Cloud Storage buckets or BigQuery datasets. Supported formats for Cloud Storage buckets include Parquet, Avro, and ORC.

Discovery configuration
Discovery is enabled by default when you create a new zone or asset. You can disable Discovery at the zone or asset level.

For each Dataplex asset with Discovery enabled, Dataplex does the following:

Scans the data associated with the asset.
Groups structured and semi-structured files into tables.
Collects technical metadata, such as table name, schema, and partition definition.
For unstructured data, such as images and videos, Dataplex Discovery automatically detects and registers groups of files sharing media type as filesets. For example, if gs://images/group1 contains GIF images, and gs://images/group2 contains JPEG images, Dataplex Discovery detects and registers two filesets. For structured data, such as Avro, Discovery detects files only if they are located in folders that contain the same data format and schema.

Reference : https://cloud.google.com/dataplex/docs/discover-data#exclude-files-from-Discovery

While JSON and CSV can technically be stored in curated zones, it is not a common practice due to the reasons mentioned above. no where in the mention link its mention that there is a restriction.

While none of the original options (A, B, C, or D) directly address the issue, the closest solution is:

Move the JSON and CSV files to a raw zone. (This was previously marked as the most voted option, but it's not ideal due to data organization disruption)
Here's why this approach might be necessary (but not ideal):

Dataplex curated zones currently don't support native processing of JSON and CSV formats. They are designed for structured data formats like Parquet, Avro, or ORC.

Option A
https://cloud.google.com/dataplex/docs/add-zone#raw-zones

Raw zones are the only zones that support CSV & JSON

Its B guys, i encounter this in my job, and I had to do B to make it work

GCP001 agree with him

The answer can be found reading a common config of Dataplex in this URL: https://medium.com/google-cloud/google-cloud-dataplex-part-1-lakes-zones-assets-and-discovery-5f288486cb2f

Dataplex does not allow users to create CSV files within a “curated zone”

According to this URL: https://cloud.google.com/dataplex/docs/discover-data, the auto-discovery can support CSV and Json in both Raw-Zone and Curated-Zone. I also open a console the verify it, both Raw and Curated zone can set up csv&json auto-discovery.

Discovery raises the following administrator actions whenever data-related issues are detected during scans : Inconsistent data format in a table. For example, files of different formats exist with the same table prefix. Inconsistent data format in a table. For example, files of different formats exist with the same table prefix.

- Prioritizes Data Privacy: It protects sensitive information by masking it, reducing the risk of exposure in case of unauthorized access or accidental leaks.
- Reduces Data Sensitivity: Masking renders sensitive data unusable for attackers, even if they gain access to it.
- Preserves Data Utility: Masked data can still be used for consumer analyses, as patterns and relationships are often preserved, allowing meaningful insights to be derived.

this is repeated question.

If I had to choose...

I choose C or A... A can still leave partial sensitive data available.

What made me decide on A instead of C was the "The data will be used to create consumer analyses" sentence. Having all the PIIs completely redacted from the records, we were unable to distinguish between the individual customers.

Option A, agree with raaad explanation

A. Use Dataflow and the Cloud Data Loss Prevention API to mask sensitive data. Write the processed data in BigQuery.

It's C. "A" removes data and retaining all is a requirement.

Not A since "while also retaining all of the data" is required.

Removing data will lead to unability to do study & consumer analyses. since it's likely all of consumer data is under PII.

option C
we can simply mask the data and process in biguery

C is correct starting 2020, as BigQuery come with table level access control
https://cloud.google.com/blog/products/data-analytics/introducing-table-level-access-controls-in-bigquery

B/E: Even if now BQ offers table level access control, since the number of tables can be expected to be high, controlling access at the dataset level would ease operations. That is why I would still go for E instead of C.

Less effort to assign access by dataset, not by table or view.

C doesn't make sense, if you have already selected B

If I choose Option E, then Option C and D are eliminated because, Once you provide the access level in dataset for the use group, it applies for both Table and view.

Now for remaining options A and B.
The Question itself has stated to include the table region wise in dataset, So Option A is eliminated.
So, B and E are Correct answers.

BE... If we've already split the tables into regions via datasets then why give regional access through tables again.. If not, then AC but definitely not BC.. Please help

Answer : BE

B. Ensure each table is included in a dataset for a region.
This means that you should organize your data in BigQuery into separate datasets, one for each region. Each dataset contains the tables specific to that region. This ensures that data is segregated by region.
E. Adjust the settings for each dataset to allow a related region-based security group view access.

B. Ensure each table is included in a dataset for a region.

This means that you should organize your data in BigQuery into separate datasets, one for each region. Each dataset contains the tables specific to that region. This ensures that data is segregated by region.
E. Adjust the settings for each dataset to allow a related region-based security group view access.

You should set the access controls at the dataset level in BigQuery. This means configuring access permissions for each dataset based on regional security groups. This way, you can enforce the regional access policy to the data, ensuring that users from different regions can only access the data associated with their region.
Option A is not necessary because you don't need to include all the tables in a global dataset. Segregating data into region-specific datasets is a better approach for enforcing access controls.

Options C and D are not typical actions in BigQuery. Access control and permissions are usually managed at the dataset level, and you can grant access to specific groups at that level.

It's B and E or B and C. However, B and E makes some more sense because if you have one dataset for each region and they need to access the data for each region then why not allow them access to the whole dataset? What if you want to add other supplementary tables later? If you did that on a table level you would have to add access to every table separately.

Still, I think both are valid because we don't have any extra requirement, but B E makes more sense.

The intended answer was for sure BE. If C or D would be the right answers there is absolutely no reason to do B, right? Why should you put each table into separate dataset if you then set the accesss on table/view level? What is more the question is about tables not views, so I have no idea why would anybody take D.
The issue is that this question is out of date and now the right answer would be sole C.

The two actions that should be taken are B and E.
B. Ensure each table is included in a dataset for a region: By creating separate datasets for each region and including only the tables associated with that region, you can enforce the regional access policy.

E. Adjust the settings for each dataset to allow a related region-based security group view access: By adjusting the settings for each dataset to allow only the related region-based security group view access, you can ensure that employees can only view data associated with their region.

A is incorrect because including all tables in a global dataset would not enforce the regional access policy.

C is incorrect because adjusting the settings for each table is not a scalable solution, especially as the number of tables grows.

D is incorrect because adjusting the settings for each view does not ensure that employees can only view data associated with their region.

B: Location is on dataset level: https://cloud.google.com/bigquery/docs/datasets#dataset_limitations
C: IAM can be set on table level

Guys,
there are 2 possible combinations
If you think that each table represents a region, then they should all be in a global dataset and you should apply table access control to them.
So A+C

Otherwise you would put each table in a regional dataset, and apply access control to the dataset. Why would you create a dataset for the purpose of controlling regional access, and then only apply the controls to a table inside it? that is not extensible in the future.
Anyway create dataset+access control for dataset (B+E) is also valid.

Which to choose? I dont know.

First put tables in region-dedicated dataset (B)
Then, ensure access control at dataset level (by creating region-based security groups) (E)

I would vote for AC. As we already split the table for each region, why do we need to split the dataset per region? Furthermore, the access control will be provided to the users based on table level anyway.

if you create table-level access control and grant it to different groups for different tables, what is the point of putting tables in different datasets and different regions?
So i choose BE

BigQuery come with table level access control. Since we can have table-level access and each region represents a table, B & C is correct answer.

- Decentralized ownership: Each department controls its data lake, aligning with the core principle of data ownership in a data mesh.
- Self-service data access: Departments can create and manage their own Cloud Storage buckets and BigQuery datasets within their data lakes, enabling self-service data access.
- Interdepartmental sharing: Dataplex facilitates data sharing by enabling departments to publish their data products from their data lakes, making it easily discoverable and usable by other departments.

It says we want to work with GCS data assets, thus Dataplex is a better option, so it's option D!

For those who says B as correct answer, I'd say that question states data assets in both GCS and BQ, however Analytics Hub is focused primarily for BQ assets.

D because it looks like B is impossible due to the lack of GCS support in Analytics Hub

In "a data mesh approach to share the data between sales, product design, and marketing departments", Analytics Hub is the solution.

B is better option as organization is migrating to google cloud, that means teams doesnt have much hands on, analytical hub is more ease to use and solved the purpose as compared to dataplex were setup itself if very complex.

For a straightforward data mesh approach where the focus is on decentralizing data management while enabling easy data sharing and discovery, Analytics Hub is often the more appropriate choice due to its simplicity and directness. It facilitates the core objectives of a data mesh—decentralized data ownership and accessible data sharing—without the added complexity of managing data lakes and advanced governance features.

I think its B. I know since we are talking about Datamesh we want to go to the Dataplex service suddenly. However, in Dataplex a Lake can only have assets (bq tables etc) that are in the same project as the Dataplex service.

Example: There is bq table in project A and B. I want to to create a Lake in Dataplex in Project A that contains tables of project B. I can´t do that, i can only host tables of the Project A, since the Lake is in project A.

With this said, I think the best option is B, because the datamesh approach is related to "to share the data between sales, product design, and marketing departments". So the question is focusing only in the sharing part of the datamesh. Option B fits just fine.

Option D

that's pure data mesh, which is what dataplex has been built for

For me, Dataplex looks more logical

D. Dataplex looks more suitable for data mesh approach, Check the ref - https://cloud.google.com/dataplex/docs/introduction

✅[A] is the only acceptable answer.
❌[B] rejected (no need to elaborate)
❌[C] and [D] rejected. Why should we be obliged to use Cloud Storage? Other storage options in Google Cloud aren't compliant with "major data protection standards"?
=============================================
❗[D] has another rejection reason, the following quotes:
🔸From <https://cloud.google.com/iam/docs/service-accounts>: "You can add service accounts to a Google group, then grant roles to the group. However, adding service accounts to groups is not a best practice. Service accounts are used by applications, and each application is likely to have its own access requirements"
🔸From <https://cloud.google.com/iam/docs/best-practices-service-accounts#groups>: "Avoid using groups for granting service accounts access to resources"

for A: please refer to this link below which suggests "Sharing a single service account across multiple applications can complicate the management of the service account" - meaning it's not a best practice.
https://cloud.google.com/iam/docs/best-practices-service-accounts#single-purpose
Also, what if we have hundreds of users, does it really make sense to manage each user's IAM individually?


for D: it's indeed not one of the best practices but I believe it's much more managable and better than A

Without Cloud storage, I believe just DLP does not provide encryption. DLP can redact or mask data, not encrypt it. Only on Cloud storage, encryption can be performed. So seems like option D is the closest choice, though service accounts should NOT be attached to IAM groups.

To align with Google's recommended practices for managing access to personally identifiable information (PII) in compliance with banking industry regulations, let's analyze the options:

A. Assign the required IAM roles to every employee, and create a single service account to access project resources: While assigning specific IAM roles to employees is a good practice for access control, using a single service account for all access to PII is not ideal. Service accounts should be used for applications and automated processes, not as a shared account for multiple users or employees.

Option D - Using multiple service accounts attached to IAM groups helps enforce the principle of least privilege. Each group can be assigned only the necessary permissions, reducing the risk of unauthorized access to sensitive data.

Google Cloud Storage is designed to comply with major data protection standards. Creating multiple service accounts and attaching them to IAM groups provides granular control over who has access to the data. This approach is aligned with the principle of least privilege, a security best practice where a user is given the minimum levels of access necessary to complete their tasks.

It´s not A because
1. assigning IAM roles to single users instead of groups is not Google best practice, and
2. the question explicitly states that we want to use multiple service accounts.

D. Use Cloud Storage to comply with major data protection standards. Use multiple service accounts attached to IAM groups to grant the appropriate access to each group.

- Google Cloud Storage is built for secure and compliant data storage. It supports compliance with major data protection standards, which is essential in the banking industry where data protection regulations are stringent.
- Service accounts in Google Cloud represent non-human users (applications or services) that need to authenticate and be authorized to access specific Google Cloud resources.
- Creating multiple service accounts attached to IAM groups allows you to manage access control in a granular manner. This follows the principle of least privilege, providing each group with only the permissions they need to perform their tasks, which is a recommended practice for managing access to sensitive data like PII.

Why are so many questions like this?
None of the answers is best practice.

I could be wrong but I think the wording in D caused this confusion, so it is an English problem. -- "Use multiple service accounts attached to IAM groups to grant the appropriate access to each group"
I believe what D really means is that you can create a group for a bunch of people who only need access to resource A, so attach a Service account to the group and service account only have access to A.
Then you create another group for another bunch of people who only need access to resource B, so attach a service account to this group. this service account can only access to B.
So each group/service account has a very specific access target, and purpose of the group is very narrowly defined which is allowed by best practice. However, wording in option D merged all these into one sentence causing confusions.
Option A is an administrative nightmare to manage IAM for a larger user population which is actually also against GCP best practices.

D it is

D is right

A is the correct one

Agree with NicolasN, D is bad practice. For D this may result in permission creep, where a group is granted access to an increasing number of resources. Only grant service accounts specific access to resources.

A is the answer, as NicalsN says.
https://cloud.google.com/iam/docs/service-accounts#groups

Avoid using groups for granting service accounts access to resources -> D

Why GCS?

- The constraints/gcp.resourceLocations organization policy constraint is used to define where resources in the organization can be created.
- Setting it to in:europe-west3-locations would specify that resources can only be created in the europe-west3 region.

https://cloud.google.com/resource-manager/docs/organization-policy/defining-locations#location_types

B, D
B - Increase the Cloud Composer 2 environment size from medium to large.
Increasing the environment size will provide more resources (including memory) to the entire environment, which should help mitigate memory usage issues. This will also support scaling if the jobs demand more resources.
D. Increase the memory available to the Airflow workers.
Increasing memory for Airflow workers will directly address the memory usage issue that's causing the pod evictions. By allocating more memory to the workers, they can handle larger tasks or more intensive workloads without failing due to memory constraints.

Option A

Option A

Set the constraints/gcp.resourceLocations organization policy constraint to in:europe-west3-locations.

Create a Dataplex lake that acts as the domain for your data mesh.
Add zones to your lake that represents individual teams within each domain and provide managed data contracts.
Attach assets that map to data stored in Cloud Storage.
https://cloud.google.com/transfer-appliance/docs/4.0/overview?_gl=1*8pbq1*_up*MQ..&gclid=CjwKCAiAg8S7BhATEiwAO2-R6gOYtlc2FJa7zE4lhz3-2f00x9F3hwgul9lYjfJs2cAprxOIeXq_NhoCw-8QAvD_BwE&gclsrc=aw.ds

just like MaxNRG said

Answer D

D. 1. Create a Dataplex virtual lake for each data product, and create multiple zones for landing, raw, and curated data.
2. Provide the data engineering teams with full access to the virtual lake assigned to their data product.

Lake: A logical construct representing a data domain or business unit. For example, to organize data based on group usage, you can set up a lake for each department (for example, Retail, Sales, Finance).
Zone: A subdomain within a lake, which is useful to categorize data by the following:
Stage: For example, landing, raw, curated data analytics, and curated data science.

D: 1 virtual lake per Data Product (which stands for domain basically), zones to split data by "status". Each Data Eng team can access their own data exclusively and in a data mesh compliant way

The best approach is to create a Dataplex virtual lake for each data product, with multiple zones for landing, raw, and curated data. Then provide the data engineering teams with access only to the zones they need within the virtual lake assigned to their product.

To enable teams to easily share curated data products, you should use cross-lake sharing in Dataplex. This allows curated zones to be shared across virtual lakes while maintaining data isolation for other zones.

I believe the answer is B, but there is a misspelling in the answer, should say "create multiple zones"

Each lake should be created per data product since data product sounds like a domain in this question.

Since we have landing, raw, curated data, we should create different zones.

"Zones are of two types: raw and curated.

Raw zone: Contains data that is in its raw format and not subject to strict type-checking.

Curated zone: Contains data that is cleaned, formatted, and ready for analytics. The data is columnar, Hive-partitioned, and stored in Parquet, Avro, Orc files, or BigQuery tables. Data undergoes type-checking- for example, to prohibit the use of CSV files because they don't perform as well for SQL access."

Ref: https://cloud.google.com/dataplex/docs/introduction#terminology

why not B?

Why not B?

The answer is D

Virtual Lake per Data Product: Each virtual lake acts as a self-contained domain for a specific data product, aligning with the data mesh principle of decentralized ownership and responsibility.
Team Autonomy: Teams have full control over their virtual lake, enabling independent development, management, and sharing of their data products.

B. While C and D are intriguing, they don't specify how to enable customer service representatives to receive access to the encryption token.

I don't see why we should use DLP since we know exactly the column that should be locked or encrypted. On the other hand having a cryptographic representation of SSN helps to aggregate/analyse entries. So I will vote for D, but B is much more easy to implement. Garbage question indeed.

In the question, there is no mention of SSN column.

Authorized users can decrypt the FPE tokens back to the original GIINs, D is the best option.

D (FPE) does indeed allow encryption to be reversed if desired, allowing operatives to review the original key. This makes it preferable to B, as it's also more secure.

D:
SSN is only tied to USA not in any other countries, The question did not mention SSN.

The best option is D - Before loading the data into BigQuery, use Cloud Data Loss Prevention (DLP) to replace input values with a cryptographic format-preserving encryption token.

The key reasons are:

DLP allows redacting sensitive PII like SSNs before loading into BigQuery. This provides security by default for the raw SSN values.
Using format-preserving encryption keeps the column format intact while still encrypting, allowing business logic relying on SSN format to continue functioning.
The encrypted tokens can be reversed to view original SSNs when required, meeting the access requirement for customer service reps.

Even if you provide Column level access control, The Data Owners or other hierarchies above it will also be able to view very sensitive data. Better to just use encryption and decryption. As this data can also never be used for any analytic workloads

Answer has to be D. Question says "you decide to redact your customers' Government issued Identification Number while allowing customer service representatives to view the original values when necessary"... Redact... view the original values... D is the only choice.

It might not be D!
Since - only the Frame is kept. the data will be changed.
Format Preserving Encryption (FPE), endorsed by NIST, is an advanced encryption technique that transforms data into an encrypted format while preserving its original structure. For instance, a 16-digit credit card number encrypted with FPE will still be a 16-digit number

Customer service needs to see the original value, not possible with other options.

of course B

I believe the crux to the question is that the cryptographic format-preserving encryption token is re-identifiable, whereas the cryptographic hash is not: https://cloud.google.com/dlp/docs/transformations-reference

Therefore, customer service can view the original values when necessary in case of D.

the question mentions that "user data is sent to Pub/Sub before being ingested" instead of just saying data goes to big query through pub/sub. So some alteration is expected before being injected into the big query. So option D should work.

D. The question says giving CSR's access to values "when necessary" - not default access like given in B. D is a better option using the token.

One of the key requirement is to be able to let authorized personel see the ID. D doesn't specify that.

The answer is between B and D as well described in many comments.

I personally do not see any reason to keep the information available using a token or a mask. It is not a PAN card number, it's just a personal ID. It should not be useful for analytical purposes.

I'm gonna go for D then

- each domain should manage their own lake’s data assets

vote C

Option C

Option C - create a lake for each domain, each team manages its own assets

Separate lakes for each team
Zones within each lake dedicated to different domains

C.
1. Create one lake for each domain. Inside each lake, create one zone for each team.
2. Attach each of the BigQuery datasets created by the individual teams as assets to the respective zone.
3. Direct each domain to manage their own lake’s data assets.

- BigLake Integration: BigLake allows you to define tables on top of data in Cloud Storage, providing a bridge between data lake storage and BigQuery's powerful analytics capabilities. This approach is cost-effective and scalable.
- Data Catalog for Governance: Creating a taxonomy of policy tags in Google Cloud's Data Catalog and applying these tags to specific columns in your BigLake tables enables fine-grained, column-level access control.
- Processing with Spark and SQL: The Spark-BigQuery connector allows data scientists to process data using Apache Spark directly against BigQuery (and BigLake tables). This supports both Spark and SQL processing needs.
- Scalability into a Data Mesh: BigLake and Data Catalog are designed to scale and support the data mesh architecture, which involves decentralized data ownership and governance.

Going with 'B' based on the comments

Option B, agree with comments explanation

BigLake leverages existing Cloud Storage infrastructure, eliminating the need for a dedicated Dataproc cluster, reducing costs significantly.

C.
1. Load the data to BigQuery tables.
2. Create a taxonomy of policy tags in Data Catalog.
3. Add policy tags to columns.
4. Process with the Spark-BigQuery connector or BigQuery SQL.

The cloud function with DLP seems the best option

DLP is required

D option

D. Build a Cloud Function that reads the topics and makes a call to the Cloud Data Loss Prevention (Cloud DLP) API. Use the tagging and confidence levels to either pass or quarantine the data in a bucket for review.

D. Dataplex

Option D, no doubt

Option D, no doubt

Option D

Clearly D

Agree with Dataplex option

Straight forward

D. Use Dataplex to manage data, track data lineage, and perform data quality validation.

Format-preserving encryption (FPE) with FFX in Cloud DLP is a strong choice for de-identifying PII like email addresses. FPE maintains the format of the data and ensures that the same input results in the same encrypted output consistently. This means the email fields in both datasets can be encrypted to the same value, allowing for accurate joins in BigQuery while keeping the actual email addresses hidden.

As it states "You need to de-identify the email field in both the datasets before loading them into BigQuery for analysts" data masking should not be an option as the data would stored unmasked in BigQuery?

Option A:
Masking: Simple masking might not preserve the uniqueness and joinability of the email field, making it difficult to perform accurate joins between datasets.
Option C and D:
Dynamic Data Masking: These options involve masking the email field dynamically within BigQuery, which does not address the requirement to de-identify data before loading into BigQuery. Additionally, dynamic masking does not prevent access to the actual email data before it is loaded into BigQuery, potentially exposing PII during the data ingestion process.

format-preserving encryption with FFX is required as the analysts want to perform JOINs

Option B
https://cloud.google.com/sensitive-data-protection/docs/pseudonymization

A) masking = replace with a surrogate character like # or * = output not unique, so cannot apply joins
C and D) question specifies to de-identify BEFORE loading into BQ, whereas these options perform dynamic masking IN BigQuery.

Therefore, only valid option is B.

Option C. The need is to just mask the data to Analyst, without modifying the underlying data. Moreover, it's stored on 2 separate tables and the analysts need to be able to perform joins based on the masked data. Dynamic masking is the right module and the right masking rule is email mask (https://cloud.google.com/bigquery/docs/column-data-masking-intro#masking_options) which guarantees the join capabilities join

A wouldn't preserve the email format
C&D maskedReader roles still grant access to the underlying values.
the only option is B

I will go for C, because there is a separate type of masking for emails, so whe to use the dafault? https://cloud.google.com/bigquery/docs/column-data-masking-intro#masking_options

data masking with BQ is correct with email masking rule.
Ref - https://cloud.google.com/bigquery/docs/column-data-masking-intro

why not B?

- The reason option C works well is that dynamic data masking in BigQuery allows the underlying data to remain unaltered (thus preserving the ability to join on this field), while also preventing analysts from viewing the actual PII.
- The analysts can query and join the data as needed for their analysis, but when they access the data, the email field will be masked according to the policy tag, and they will only see the masked version.

D. 1. Load the CSV files from Cloud Storage into a BigQuery table, and enable dynamic data masking.
2. Create a policy tag with the default masking value as the data masking rule.
3. Assign the policy to the email field in both tables.
4. Assign the Identity and Access Management bigquerydatapolicy.maskedReader role for the BigQuery tables to the analysts

- dataplex.dataOwner: Grants full control over data assets, including reading, writing, managing, and granting access to others.
- dataplex.dataReader: Allows users to read data but not modify it.

The quetion is for BigQuery AND Cloud Storage for a Data Lake, so you should assign IAM permissions for both of them. C is correct.

Option A

A correct answer

Option A clearly correct

A.
1. Grant the dataplex.dataOwner role to the data engineer group on the customer data lake.
2. Grant the dataplex.dataReader role to the analytic user group on the customer curated zone.