Here is an interesting error that we recently encountered with one of our Redshift Serverless and Redshift Provisioned clusters. We have a data sharing setup where the serverless DB is the producer cluster of certain key tables. We share these tables to a provisioned Redshift cluster via data sharing.
When querying this particular table on the provisioned cluster through the data share with python(psycopg2) and airflow we received the following error.
|
1 2 3 4 5 6 7 8 |
2024-04-29 18:06:25,073 - ERROR - | Query aborted due to read failure on a perm block. | HINT: Please try again. 2024-04-29 18:06:25,073 - ERROR - | Stacktrace: 2024-04-29 18:06:25,073 - ERROR - | Traceback (most recent call last): | File "/usr/local/airflow/.local/lib/python3.10/site-packages/soda/execution/query/query.py", line 122, in fetchone | cursor.execute(self.sql) | psycopg2.errors.IoError: Query aborted due to read failure on a perm block. | HINT: Please try again. |
We opened a support case with AWS and was informed that this is due to a meta data mismatch that can be resolved by running an update against the shared table on the producer side. After running this update we were back in business and things operated as normal.
The following query can be executed on the producer cluster as a mitigation: “UPDATE
SET = 1 WHERE false;” where TABLE_NAME is the name of the table on which queries are failing and COLUMN_NAME is the name of any column in this table. This query will not result in any actual change to the producer’s data, but will result in synchronizing the metadata pertaining to TABLE_NAME on the consumer and thus letting subsequent datasharing queries go through successfully.
|
1 |
"UPDATE <TABLE_NAME> SET <COLUMN_NAME> = 1 WHERE false;" |
— Jason Ralph