Skip to content

Commit

Permalink
[SPARK-47750][DOCS][SQL] Postgres: Document Mapping Spark SQL Data Ty…
Browse files Browse the repository at this point in the history
…pes to PostgreSQL

### What changes were proposed in this pull request?

This PR added user documentation for Mapping Spark SQL Data Types to PostgreSQL

### Why are the changes needed?

doc improvement
### Does this PR introduce _any_ user-facing change?

no

### How was this patch tested?

doc build
![image](https://github.com/apache/spark/assets/8326978/2745aa86-67ef-49ee-bdd2-a49eea800851)

### Was this patch authored or co-authored using generative AI tooling?

Closes apache#45912 from yaooqinn/SPARK-47750.

Authored-by: Kent Yao <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
  • Loading branch information
yaooqinn committed Apr 8, 2024
1 parent eb8e997 commit f0d8f82
Showing 1 changed file with 182 additions and 0 deletions.
182 changes: 182 additions & 0 deletions docs/sql-data-sources-jdbc.md
Original file line number Diff line number Diff line change
Expand Up @@ -1009,3 +1009,185 @@ as the activated JDBC Driver. Note that, different JDBC drivers, or different ve
</tr>
</tbody>
</table>


### Mapping Spark SQL Data Types to PostgreSQL

The below table describes the data type conversions from Spark SQL Data Types to PostgreSQL data types,
when creating, altering, or writing data to a PostgreSQL table using the built-in jdbc data source with
the [PostgreSQL JDBC Driver](https://mvnrepository.com/artifact/org.postgresql/postgresql) as the activated JDBC Driver.


<table>
<thead>
<tr>
<th><b>Spark SQL Data Type</b></th>
<th><b>PostgreSQL Data Type</b></th>
<th><b>Remarks</b></th>
</tr>
</thead>
<tbody>
<tr>
<td>BooleanType</td>
<td>boolean</td>
<td></td>
</tr>
<tr>
<td>ByteType</td>
<td>smallint</td>
<td></td>
</tr>
<tr>
<td>ShortType</td>
<td>smallint</td>
<td></td>
</tr>
<tr>
<td>IntegerType</td>
<td>integer</td>
<td></td>
</tr>
<tr>
<td>LongType</td>
<td>bigint</td>
<td></td>
</tr>
<tr>
<td>FloatType</td>
<td>float4</td>
<td></td>
</tr>
<tr>
<td>DoubleType</td>
<td>float8</td>
<td></td>
</tr>
<tr>
<td>DecimalType(p, s)</td>
<td>numeric(p,s)</td>
<td></td>
</tr>
<tr>
<td>DateType</td>
<td>date</td>
<td></td>
</tr>
<tr>
<td>TimestampType</td>
<td>timestamp</td>
<td></td>
</tr>
<tr>
<td>TimestampNTZType</td>
<td>timestamp</td>
<td></td>
</tr>
<tr>
<td>StringType</td>
<td>text</td>
<td></td>
</tr>
<tr>
<td>BinaryType</td>
<td>bytea</td>
<td></td>
</tr>
<tr>
<td>CharType(n)</td>
<td>CHAR(n)</td>
<td></td>
</tr>
<tr>
<td>VarcharType(n)</td>
<td>VARCHAR(n)</td>
<td></td>
</tr>
<tr>
<td>ArrayType</td>
<td><table>
<thead>
<tr>
<th><b>Element type</b></th>
<th><b>PG Array</b></th>
</tr>
</thead>
<tbody>
<tr>
<td>BooleanType</td>
<td>boolean[]</td>
</tr>
<tr>
<td>ByteType</td>
<td>smallint[]</td>
</tr>
<tr>
<td>ShortType</td>
<td>smallint[]</td>
</tr>
<tr>
<td>IntegerType</td>
<td>integer[]</td>
</tr>
<tr>
<td>LongType</td>
<td>bigint[]</td>
</tr>
<tr>
<td>FloatType</td>
<td>float4[]</td>
</tr>
<tr>
<td>DoubleType</td>
<td>float8[]</td>
</tr>
<tr>
<td>DecimalType(p, s)</td>
<td>numeric(p,s)[]</td>
</tr>
<tr>
<td>DateType</td>
<td>date[]</td>
</tr>
<tr>
<td>TimestampType</td>
<td>timestamp[]</td>
</tr>
<tr>
<td>TimestampNTZType</td>
<td>timestamp[]</td>
</tr>
<tr>
<td>StringType</td>
<td>text[]</td>
</tr>
<tr>
<td>BinaryType</td>
<td>bytea[]</td>
</tr>
<tr>
<td>CharType(n)</td>
<td>char(n)[]</td>
</tr>
<tr>
<td>VarcharType(n)</td>
<td>varchar(n)[]</td>
</tr>
</tbody>
</table></td>
<td>If the element type is an ArrayType, it converts to Postgres multidimensional array. <br>For instance, <br><code>ArrayType(ArrayType(StringType))</code> converts to <code>text[][]</code>, <br><code>ArrayType(ArrayType(ArrayType(LongType)))</code> converts to <code>bigint[][][]</code></td>
</tr>
</tbody>
</table>

The Spark Catalyst data types below are not supported with suitable PostgreSQL types.

- DayTimeIntervalType
- YearMonthIntervalType
- CalendarIntervalType
- ArrayType - if the element type is not listed above
- MapType
- StructType
- UserDefinedType
- NullType
- ObjectType
- VariantType

0 comments on commit f0d8f82

Please sign in to comment.