pyspark.sql.functions.try_element_at#

pyspark.sql.functions.try_element_at(col, extraction)[source]#

Collection function: (array, index) - Returns element of array at given (1-based) index. If Index is 0, Spark will throw an error. If index < 0, accesses elements from the last to the first. The function always returns NULL if the index exceeds the length of the array.

(map, key) - Returns value for given key. The function always returns NULL if the key is not contained in the map.

Added in version 3.5.0.

Parameters:

colColumn or str: name of column containing array or map. A column that evaluates to an array or map.
extraction: index to check for in array or key to check for in map. A column that evaluates to an integer for an array, or the key type for a map.

Returns:

Column: Returns a column of the element type of the input array, or the value type of the input map.

See also

pyspark.sql.functions.get()
pyspark.sql.functions.element_at()

Notes

The position is not zero based, but 1 based index. If extraction is a string, try_element_at() treats it as a column name, while element_at() treats it as a literal string.

Examples

Example 1: Getting the first element of an array

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([(["a", "b", "c"],)], ['data'])
>>> df.select(sf.try_element_at(df.data, sf.lit(1))).show()
+-----------------------+
|try_element_at(data, 1)|
+-----------------------+
|                      a|
+-----------------------+

Example 2: Getting the last element of an array using negative index

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([(["a", "b", "c"],)], ['data'])
>>> df.select(sf.try_element_at(df.data, sf.lit(-1))).show()
+------------------------+
|try_element_at(data, -1)|
+------------------------+
|                       c|
+------------------------+

Example 3: Getting a value from a map using a key

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([({"a": 1.0, "b": 2.0},)], ['data'])
>>> df.select(sf.try_element_at(df.data, sf.lit("a"))).show()
+-----------------------+
|try_element_at(data, a)|
+-----------------------+
|                    1.0|
+-----------------------+

Example 4: Getting a non-existing element from an array

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([(["a", "b", "c"],)], ['data'])
>>> df.select(sf.try_element_at(df.data, sf.lit(4))).show()
+-----------------------+
|try_element_at(data, 4)|
+-----------------------+
|                   NULL|
+-----------------------+

Example 5: Getting a non-existing value from a map using a key

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([({"a": 1.0, "b": 2.0},)], ['data'])
>>> df.select(sf.try_element_at(df.data, sf.lit("c"))).show()
+-----------------------+
|try_element_at(data, c)|
+-----------------------+
|                   NULL|
+-----------------------+

Example 6: Getting a value from a map using a column name as the key

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([({"a": 1.0, "b": 2.0}, "a")], ['data', 'b'])
>>> df.select(sf.try_element_at(df.data, 'b')).show()
+-----------------------+
|try_element_at(data, b)|
+-----------------------+
|                    1.0|
+-----------------------+