StructField

class pyspark.sql.types.StructField(name, dataType, nullable=True, metadata=None)

A field in StructType.

Parameters

namestr

name of the field.

dataTypeDataType

DataType of the field.

nullablebool, optional

whether the field can be null (None) or not.

metadatadict, optional

a dict from string to simple type that can be toInternald to JSON automatically

Examples

>>> from pyspark.sql.types import StringType, StructField
>>> (StructField("f1", StringType(), True)
...      == StructField("f1", StringType(), True))
True
>>> (StructField("f1", StringType(), True)
...      == StructField("f2", StringType(), True))
False

Methods

fromDDL(ddl)	Creates DataType for a given DDL-formatted string.
fromInternal(obj)	Converts an internal SQL object into a native Python object.
fromJson(json)
getCollationMetadata()
getCollationsMap(metadata)
json()
jsonValue()
needConversion()	Does this type needs conversion between Python object and internal SQL object.
schemaCollationValue(dt)
simpleString()
toInternal(obj)	Converts a Python object into an internal SQL object.
typeName()

Methods Documentation

classmethod fromDDL(ddl)

Creates DataType for a given DDL-formatted string.

New in version 4.0.0.

Parameters

ddlstr

DDL-formatted string representation of types, e.g. pyspark.sql.types.DataType.simpleString, except that top level struct type can omit the struct<> for the compatibility reason with spark.createDataFrame and Python UDFs.

Returns

DataType

Examples

Create a StructType by the corresponding DDL formatted string.

>>> from pyspark.sql.types import DataType
>>> DataType.fromDDL("b string, a int")
StructType([StructField('b', StringType(), True), StructField('a', IntegerType(), True)])

Create a single DataType by the corresponding DDL formatted string.

>>> DataType.fromDDL("decimal(10,10)")
DecimalType(10,10)

Create a StructType by the legacy string format.

>>> DataType.fromDDL("b: string, a: int")
StructType([StructField('b', StringType(), True), StructField('a', IntegerType(), True)])

fromInternal(obj)

Converts an internal SQL object into a native Python object.

classmethod fromJson(json)

getCollationMetadata()

getCollationsMap(metadata)

json()

jsonValue()

needConversion()

Does this type needs conversion between Python object and internal SQL object.

This is used to avoid the unnecessary conversion for ArrayType/MapType/StructType.

schemaCollationValue(dt)

simpleString()

toInternal(obj)

Converts a Python object into an internal SQL object.

typeName()